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Executive Summary 


The project 


This report describes the development and pilot evaluation of SMART Spaces. This programme aims 
to boost GCSE science outcomes by applying the principle that information is more easily learnt when 
it is repeated multiple times, with time passing between the repetitions. This approach is known as 
‘spaced learning’ and is contrasted with a ‘massed learning’ approach, where content is learnt all at 
once with no spacing. The development of the programme was led by a team from the Hallam 
Teaching School Alliance (HTSA). SMART Spaces prepares Year 9 and 10 students for GCSE 
examinations at the end of Year 10. Teachers were trained to deliver three lessons focused on 
chemistry, physics, and biology curriculum content, which were repeated over three consecutive days. 
Pupils did an unrelated physical activity in the spaces between intensive repetitions of science 
content. Teachers received one day of training and were provided with PowerPoint slides to deliver 
during the lessons. 


The Centre for Evidence and Social Innovation (CESI) at Queen’s University Belfast (QUB) worked 
with HTSA to develop SMART Spaces, test its feasibility, and test three different approaches to 
arranging the spaced learning across the three days (see Table 1). 


This project was jointly funded by the EEF and Wellcome Trust as part of the Education and 
Neuroscience partnership. 


Key conclusions 


1. The spaced learning principle is based on relatively strong evidence and this pilot suggests that 
SMART Spaces has evidence of promise. 

2. This pilot study demonstrated that SMART Spaces can feasibly be delivered in English schools. 
Both teachers and pupils appeared to enjoy and engage with the programme. 

3. Teachers generally reported that they delivered SMART Spaces lessons as prescribed and did 
not make major alterations. 

4. The small randomised controlled trial (RCT) provided some preliminary evidence that the most 
promising approach to spaced learning combines the use of both ten-minute and 24-hour 
spaces between curriculum content. However, this was a small study and a larger trial is needed 
to better understand the impact of the programme. 

5. SMART Spaces is ready for a larger RCT to evaluate its impact on GCSE attainment. 


What are the findings? 


The principle of spaced learning is supported by evidence from two scientific fields, neuroscience and 
cognitive psychology. The neuroscience literature supports the use of shorter spaces between 
learning (of around ten minutes), and the cognitive psychology literature supports longer spaces (of 
around 24 hours). This study used a small randomised controlled trial to investigate the efficacy of 
three different versions of the SMART Spaces programme (see Table 1). This suggested that Version 
3—which combined 24-hour and ten-minute spacing—appeared to be the most promising variant. 
This version was also supported by both the neuroscientific and cognitive psychology literature, and 
will be used in future implementations of SMART Spaces. However, this study was only intended to 
provide preliminary evidence and is smaller than EEF efficacy trials. A larger trial is required before 
any firm conclusions can be drawn about the efficacy of SMART Spaces. 
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Table 1: the three versions of the SMART Spaces programme 


Version 1 (10-minute Version 2 (24-hour Version 3 (10-minute and 
spaces) spaces) 24-hour spaces) 

Day 1 12 minutes of chemistry 12 minutes of chemistry 12 minutes of chemistry 
10-minute ‘space’ 12 minutes of physics 10 minutes of ‘space’ 
12 minutes of chemistry 12 minutes of biology 12 minutes of physics 
repeated 20 minutes of ‘space’ atend | 10 minutes of ‘space’ 
10-minute ‘space’ 12 minutes of biology 
12 minutes of chemistry 
repeated 

Day 2 As day 1 but for physics As day 1 As day 1 

Day 3 As day 1 but for biology As day 1 As day 1 


By the end of this developmental pilot, HTSA had developed SMART Spaces into a programme that 
could feasibly be delivered in English schools. Both teachers and pupils gave substantial positive 
feedback about the programme. Most teachers did not feel they needed much further support to 
deliver the intervention, which suggests the training was successful. Teachers generally reported that 
they delivered SMART Spaces lessons as prescribed and did not make major alterations. The 
programme is ready for a larger trial; it is clearly defined and could be delivered to the large number of 
schools required in an efficacy trial. The evaluator estimated that the programme would cost schools 
£10 per pupil in the first year of delivery, a very low cost. Schools will also need to arrange one full 
day of cover, and this might result in further costs. 


How was the pilot conducted? 


This evaluation was designed to provide feedback to the Hallam team throughout their development 
of the programme, and, at the end of the project, provide the EEF with a judgement of the 
programme’s evidence of promise, feasibility, and readiness for trial. It was conducted over one and a 
half years and divided into three phases: 


e Inthe first phase, CESI worked with HTSA to carry out a literature review, which informed the 
design of a logic model for the programme and selection of three versions of the programme 
for later testing (see Table 1). 

e The second phase used qualitative methods and worked with four schools to examine the 
feasibility of the three programme variations. The programme was adapted in response to 
feedback from this feasibility stage. 

e The final phase aimed to provide some preliminary evidence about the effectiveness of the 
three different versions. The three programme variants were compared against a control 
group which received the PowerPoint slides, but no spacing protocol, and a control group 
which received neither slides nor spacing protocol. 


Table 2: Summary of pilot findings 
Question slave liare] Comment 


The spaced learning principle is supported by evidence 
from both the cognitive science and neuroscience 
literature. The version which combined ten-minute and 
24-hour spaces appeared to be the most promising. 
However, a larger trial is required before drawing any 
firm conclusions about the effectiveness of SMART 
Spaces. 

The programme was delivered successfully and was 
acceptable to both teachers and pupils. 
SMART Spaces is a well-defined and scalable 
programme that is ready for an efficacy randomised 
controlled trial (RCT). 


Is there evidence to support 
the theory of change? 


Was the approach feasible? 


Is the approach ready to be 
evaluated in a trial? 
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Introduction 


Intervention 


This research report details the design, feasibility, and optimisation of a GCSE science attainment 
intervention called SMART Spaces, constituting an early phase innovation or ‘proof of concept’ 
project. The main driver for this research was the mutual interest of the Wellcome Trust and the 
Education Endowment Foundation in the translation of neuroscientific evidence into teaching practice. 
The research in this report was informed by neuroscientific evidence and cognitive experimental 
psychology evidence, as well as educational practice literature in these two research areas. 


The appointed evaluation team, the Centre for Evidence and Social Innovation (CESI), and project 
team, Hallam Teaching School Alliance (HTSA), agreed at an early stage that a research and practice 
partnership would be the best approach to adopt in order to design, assess feasibility, and optimise 
the emerging programme. An iterative feedback strategy was then employed by the partners 
throughout, involving a ‘develop, test, feedback, consult, amend, and re-test’ loop. The main output of 
this working partnership, and the research conducted in this study, is the latest version of SMART 
Spaces, which is described in the TIDieR checklist below in Table 3 (a checklist detailing the training 
and materials relating to the programme, see Hoffman et al., 2014). 


Table 3: TIDieR checklist for SMART Spaces programme 


Brief Name 
SMART Spaces: Spaced-learning for Memory And Retention Teaching (GCSE 


a Science Revision Version). 


Why 
Educational programme for GCSE students primarily used for exam revision with 


2 an aim to improve science attainment for Year 9 and 10 pupils. 
What 
3 Materials: SMART Materials, PowerPoint slides, SMART Spaces manual, and 
SMART Spaces activity pack. 
Procedures: SMART Training—teachers are trained on delivery of SMART Spaces 
in a one-day training session. 
4 SMART Spaces implementation: biology, chemistry and physics topics were taught 


in the three short 12-minute lessons with 10-minute spaces between each topic; 
this process was repeated over three consecutive days (thus providing additional 
24-hour spaces between content repetitions). 
Who provided 
5 SMART Training was provided by trainers experienced in the delivery of SMART 
Spaces. Future teachers who deliver the programme will be GCSE science 
teachers who have SMART Training. The same teacher should provide the whole 
session of SMART Spaces on the three consecutive days. 


How 
6 Whole-class programme that is conducted during three normal science lessons. | 
Where 
7 SMART Training conducted in out-of-school session, and SMART Spaces 


conducted in standard GCSE classroom. 
When and how 


much 
8 The programme covers GCSE science curriculum content in a high intensity way. 
The SMART Spaces slides are set out in three 12-minute chunks of GCSE 
chemistry, physics and biology (approximately one third of each course) content to 
be taught in one-hour lessons, repeated on three consecutive days. 
Tailoring 
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9 The programme logic model was designed using neuroscientific evidence, 
cognitive psychology evidence, and educational practice literature in both areas. 
Feasibility was piloted in four schools and optimisation was achieved by trialling 
three different variants against two controls in 12 schools (detailed in this report). 
Modifications 

10 The optimisation study explored the different types of spaces (inter-study intervals) 
that could be used in delivery of SMART Spaces. It found that there was a clear 
benefit to using a combination of both ten-minute and 24-hour spaces in the 
delivery of SMART Spaces content. Some minor adaptations were made to the 
inter-study activities (for example, alternative tasks to juggling). No major 
adaptations are recommended to the emerging model with most promise of ten- 
minute and 24-hour spaces. 


How well 


Planned: effective implementation required training teachers in all twelve schools 
before they took part in the optimisation trial and all delivered their version of the 
programme. This training was planned to consist of modelling, practice, and 
feedback on programme delivery. 
12 Actual: the rationale behind this study was to look at the impact of variability in 
implementation. Among those schools who delivered the same variant there was 
no apparent difference in implementation. The content of the SMART Spaces 
programme was found to have a significant benefit over a no spaces/materials 
control in the optimisation study, with ES g = 0.19 on total scores on an attainment 
test using past GCSE questions. Pupil engagement with SMART Spaces was 
found to be a significant mediator of outcome change. 

Therefore it was deemed that SMART Spaces should include the following key 
elements: ten-minute and 24-hour intervals, SMART Training, and SMART 
Resources. 


The TIDieR checklist details the two main elements to the SMART Spaces programme—SMART 
Materials and SMART Training. The SMART Materials comprise a manual and an activities pack. The 
manual is a comprehensive guide to the SMART Spaces programme and is intended to help teachers 
deliver the programme with fidelity (that is, in a manner consistent with the original design) in any 
classroom. At the time of writing, the SMART Spaces manual is under production with a view to 
facilitating the scaling-up of the intervention in a larger trial. It essentially covers the following 
elements: background evidence relating to the programme’s development (a summary of this report), 
the programme logic model, the slides for teachers to use during the sessions (chemistry, physics and 
biology GCSE content—see example in Appendix 4), and a step-by-step guide on how to deliver the 
programme. 


The SMART Training consists of a one-day training course with an experienced trainer in the delivery 
of the programme (usually a GCSE science teacher). SMART Training is a prerequisite for all delivery 
teachers. It includes the presentation of some of the supporting evidence from neuroscience and 
cognitive psychology, but the major component is modelling how the programme is delivered, as well 
as practice and feedback for the teachers on their delivery of SMART Spaces. Specifically, the 
training schedule is: 


e the scientific background to SMART Spaces—the how and the why of why it works (20 mins); 

e how the sessions are managed, including managing the activities in the ‘spaces’ (15 mins); 

e alook at the lesson resources provided (15 mins); 

e anexperience of how a SMART Spaces session actually runs (20 mins); and 

e the opportunity to have a go at delivering a session to the other delegates with constructive 
feedback (20 mins per teacher). 
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Finally, the SMART Activities is a pack of materials that are used in the ten-minute ‘spaces’ (including 
juggling balls), and includes a description of how to conduct the activities in various classroom 
settings. 


Background evidence 
The relationship between learning and spaces in time 


Since Ebbinghaus (1885) first discovered that he could learn material from fewer repetitions when the 
repetitions were spaced apart in time rather than grouped together, the relationship between time and 
learning has fascinated everyone from school teachers to neurobiologists. ‘Spaced learning’ (also 
often referred to as ‘distributed learning’) is a learning strategy in which two or more study periods are 
separated in time by an inter-study interval (ISI). The ISI may be as brief as ten minutes, or as long as 
weeks and months. When the efficacy of a spaced learning strategy is examined, it is often compared 
with a ‘massed learning’ approach. Massed learning describes an approach where all studying or 
stimulus presentation occurs in one constant block, without any gaps in time to separate periods of 
learning. The ‘spacing effect’ refers to the benefit of spaced learning on memory and retention of 
information over massed learning. 


Evidence (reviewed below) suggests the same material can be learned more effectively using a 
spaced learning strategy rather than a massed learning strategy and, assuming that the overall 
quantity of studying is equal in time between the two approaches, this is called a spacing effect. This 
should be distinguished from the concept of a ‘lag effect’ which describes the benefit of one length of 
ISI over another for learning, when two or more spaced strategies are compared. For example, if the 
same material was learned more effectively with a 60-minute ISI between study periods than with a 
ten-minute ISI, this would be referred to as a lag effect. The comparison of different ISIs has been 
used to attempt to determine the optimal spacing strategy for different retention intervals (RI). The 
retention interval is the time between the final study period in a spaced learning procedure—or the 
single massed study period in a massed learning procedure—and the testing period. 


Another related concept to spaced learning is interleaving. In a situation where material must be 
learnt about different topics, interleaving these areas of learning, rather than blocking them together, 
may lead to better retention. For example, interleaving may follow a pattern of ‘ABCABCABC’ rather 
than an ‘AAABBBCCC’ blocked pattern of delivery. Taylor and Rohrer (2010) used a combined 
approach of spacing and interleaving for the practice of four types of mathematics problem-solving 
with children. They found that, when spacing was controlled between blocked and interleaved 
practice, interleaved practice led to higher test scores after the practice, despite the fact that 
performance on the practice day was impaired. It may be the case that interleaving leads to a more 
difficult practice experience in the short term, but improves retention in the mid to long term. 


What has past research from cognitive psychology and neuroscience told us about spaced 
learning? 


Cognitive psychology and spaced learning 


Numerous reviews of cognitive experiments on spacing effects have revealed the robustness of 
spacing effects for simple memory tasks and motor skills. These reviews also opened up the 
possibility that there is an optimal spacing interval which depends on the retention interval and the 
complexity of the task. It should be noted that, unless otherwise specified, these cognitive psychology 
reviews typically focus on adults. Lee and Genovese (1988) conducted a literature review of abstracts 
on distributed learning of motor skills published since 1968 in Psychological Abstracts, and prior to 
1968 in Perceptual and Motor Skills (1949 to 1968). They reviewed 47 articles and found that spaced 
learning improved both acquisition and retention of motor skills (mean ES = 0.91). Moss (1996) 
reviewed 120 articles on the spaced learning effect, comparing various types of learning material 
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(verbal information, intellectual skill, and motor learning). Longer spacing intervals improved the 
learning of verbal information and motor skills in over 80% of the studies reviewed. However, for more 
complex intellectual tasks (such as mathematical computation), only one third of the studies showed a 
spacing effect. 


Donovan and Radosevich (1999) reviewed 63 articles, including both experimental and quasi- 
experimental studies (mean weighted ES = 0.46) and found that the largest spacing effects were 
found in studies of learning low-complexity tasks (Such as typing). They also found that, for verbal 
tasks, extending the space improved recall up to a point, and then began to negatively affect recall if 
the space was extended too much. Janiszewski et al. (2003) reviewed 97 articles on spacing effects 
and space lengths for various types of task. The authors included studies on the basis that they 
reported a spacing effect, and included enough statistical information to calculate an effect size. They 
found that numerous factors resulted in larger spacing effect sizes including: longer spaces (mean ES 
= 0.57), complexity of stimuli (mean ES = 0.42), less meaningful stimuli (mean ES = 0.51), and 
intentional (rather than incidental) learning (mean ES = 0.35). Cepeda et al. (2006) reviewed 317 
spacing effect experiments in 184 articles, including children, adults and older adults. These were all 
verbal memory learning tasks, followed by a recall test. The authors included both an analysis of 
spaced versus massed learning, and a comparison of different lengths of spacing intervals. All but 12 
of the 217 studies showed a benefit of spaced learning over massed study, but it is important to note 
these studies reported only improvements in accuracy (due to insufficient effect size data). Cepeda et 
al. also found that increasing the spacing interval increased recall, but too long a spacing interval for a 
given retention period reduced recall (in agreement with Donovan and Radosevich, 1999). Cepeda et 
al. determined that the spacing interval during studying should increase as retention interval increases 
to optimise recall. When the retention interval was less than one minute, optimal space was also less 
than one minute. At least a one-month space is necessary for optimal recall after six months. This 
was further explored by Cepeda et al. (2009), who asked 215 undergraduates in the U.S.A. to learn 
Swahili-English pairs in one experiment, and pairs of facts and objects in another. Participants were 
randomly assigned to one of six groups, each group being assigned a different ISI, varying from five 
minutes to 14 days for the first experiment, and 20 minutes to six months for the second. They also 
examined retention intervals of ten days and six months. It was found that increases in the spacing 
interval led to a sharp improvement in retention, but if the spacing interval became too long, a slight 
decrease in retention was observed. For a six-month retention interval, it was found that a 28-day 
space was optimal. 


The body of cognitive psychology literature provides strong evidence for the benefits of spaced 
learning and for the model of an optimal ISI for a given retention interval in which the ISI is 
appropriately long in relation to the length of the RI. At this point we should note that what happens 
during the space may also influence the success of the learning. Perhaps most pertinent is the role of 
sleep. Bell et al. (2014) asked participants to learn Swahili-English word pairs and compared massed 
learning versus three types of spaced learning (12-hour space without sleep versus 12-hour space 
with sleep versus 24-hour space). Participants (n = 141) were randomly assigned to one of the four 
conditions. A spacing effect was found for all the spacing conditions, but interestingly, the two 
conditions with sleep (12 hours with sleep and 24 hours) showed a stronger advantage over massed 
learning than the 12 hours without sleep condition. It is possible that longer ISIs themselves are not 
solely responsible for their benefits over shorter ISIs, and that the sleep associated with longer 
intervals plays a role. 


Overall, it is apparent from the reviews of cognitive psychology experiments that spacing effects can 
be consistently found, that long spaces of a day or more are increasingly beneficial for learning, and 
that sleep during the space may be important for the success of memory formation. These three 
facets of the evidence provide a solid base for the design of a 24-hour space approach to spaced 
learning. 


Education Endowment Foundation 9 


Spaced Learning: the design, feasibility and optimisation SMART Spaces 


Neuroscience and spaced learning 


Studies within the field of neuroscience have explored the neurobiological basis of the spacing effect. 
Some of these have involved removing cells from animal brains (post-mortem) and stimulating them 
with either neurotransmitter chemicals or electricity. This is to mimic the input of stimulation from 
learning material to a live brain. Then the response of the brain cells is recorded, as a proxy of the 
sort of response that might happen during actual memory formation. 


These studies have focused on recording neurobiological indicators of ‘long term potentiation’ (LTP)— 
a process through which synapses may become stronger after being stimulated, and thus transmit a 
long lasting signal between neurons. Hebbian theory states that cells which are stimulated together 
form a long lasting connection and this is thought to be one mechanism for how memories are formed 
in the brain. This theory is often abridged to ‘cells which fire together, wire together’. LTP is often 
investigated as a potential biological basis for the formation of long-term memories. Electrical activity 
of synapses, proteins, and gene presence can be recorded in a cell as an indicator of LTP, and 
therefore as a potential precursor to the formation of long-term memory. 


Mauelshagen, Sherff and Carew (1998) exposed synapses removed from Aplysia (marine molluscs) 
to serotonin in either five bursts of five minutes with 15-minute intervals, or one long massed 
exposure of 25 minutes. They found that electrical responses to stimulation 24 hours later (which is 
representative of the type of cell activity important for long-term memory formation) was greater for 
spaced stimulation than massed stimulation. Fields (2009) reported studies where synapses were 
taken from sections of rat hippocampi. The hippocampus is where consolidation from short-term to 
long-term memory is thought to take place. Fields and his colleagues studied electrical activity and 
the presence of genes in these rat brain cells after electrical stimulation, as a measure of long-term 
potentiation. Fields found that the rat synapses produced twice the voltage of electrical activity after 
being stimulated in three bursts with ten-minute spaces than when they were stimulated in a massed 
pattern. They also found the presence of the protein CREB and a gene called zif268 after spaced 
stimulation, both of which we know are associated with the conversion of short- to long-term memory. 
These studies demonstrate the neurobiological processes involved in the cognitive advantages of 
spaced learning in terms of electrical activity, proteins, and genetic indicators of memory formation. 


As seen in the review of the cognitive psychology literature, the neuroscience literature also points to 
the benefit of increasing the space length for better retention. Kramar et al. (2012) used a similar 
experimental design to Fields (2009), stimulating rat brain cells with electricity but with the addition of 
exploring the effects of different spacing intervals—10, 30 and 60 minutes between periods of 
stimulation. Twice as much long-term potentiation was produced when a 60-minute interval was used 
rather than 10 or 30 minutes. Zhang et al. (2011) found that a spacing protocol including a 30-minute 
space in the stimulation of mollusc brain cells led to greater activity than the shorter spaces used in 
the Mauelshagen et al. (1998) study. Overall, the animal brain studies reveal that there is physical 
evidence of long-term memory formation being enhanced by spaced stimulation, and more recently 
that longer intervals than originally suggested by Fields (2009) may be optimal. 


Much less information is available about human brain activity during spaced learning, although some 
new developments are beginning to emerge. Earlier neuroscience research with humans did not 
actually involve explicit spaced learning tasks. Van Strien et al. (2007) had 22 adults judge if they had 
heard a word before while being presented with a list of repeated words. Recall was better for words 
which had been presented with spaced repetitions than after the massed repetitions. During the task, 
electroencephalography (EEG) recordings were made. EEG involves the recording of electrical 
activity at the scalp, which allows the size of event-related potentials (ERP, measured in voltage) to 
be used as an index of the brain’s response. Van Strien et al. (2007) found a larger change in event- 
related potentials associated with memory search and template matching (N400 and LPC) in 
response to massed rather than spaced presentations. This means that a pattern in electrical activity 
which may be indicative of memory activity showed evidence of increased difficulty performing a 
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memory task when massed presentation was used rather than spaced presentation. This suggests 
that massed presentations resulted in more difficulty performing these two aspects of recall. 


To date, we are aware of only two studies involving an explicit spaced learning task and the recording 
of human brain activity. Mollison and Curran (2011) compared the paired learning of nouns and 
pictures from two repetitions presented in either a massed or spaced format (12-second ISIs). They 
found that massed learning resulted in weaker brain responses when the items were presented the 
second time in comparison with the first time. Spaced learning did not show this suppression of brain 
responses. Furthermore, the spaced items were remembered with higher accuracy than the massed 
items. This result may indicate that attention to repeated items is better when a spacing strategy is 
used. Functional magnetic resonance imaging (fMRI) has also been used to explore spaced learning. 
Xue et al. (2012) asked adults to memorise 120 novel faces, some of which were repeated in a 
massed manner, and some in a spaced manner. Spaced learning led to more activity in a brain area 
associated with face recognition (bilateral fusiform gyrus) than massed learning. Spaced learning also 
led to better learning of items than massed learning. Again, this indicates that spaced learning leads 
to less suppression of learning responses for repetitions than massed learning. 


The human-based neuroscience studies help explain the benefits of spaced learning in terms of 
increased attention to repeated items. Due to the highly technical and time-consuming nature of 
human neuroscience studies, we have not yet seen the recording of human brain responses to long 
spacing intervals. The early neuroscience work with animals showed spacing effects with ten-minute 
spaces originally, but there is emerging evidence of the benefits of longer spaces over short. 


Spaced learning in educational settings 


Although the reviews summarised above show that the spacing effect is highly robust, some of the 
results suggest that for more complex tasks, spacing may provide less of a benefit than it does for 
simple tasks. However, more recently, progress has been made in determining that spacing effects 
can be observed even for the complex learning in normal educational practice. For example, Miles 
(2014) applied the evidence of the spacing effect from cognitive psychology in the classroom and 
examined the effect of spacing on the teaching of English grammar to South Korean undergraduate 
TESOL students, randomly assigning students to massed or spaced conditions in an experimental 
study. He compared massed against spaced learning (ISIs of one week, then four weeks, with a 
retention interval of five weeks). This study found a strong spacing effect, despite the complex nature 
of the learning task. Again, we see the benefits of longer intervals between study periods for a long 
retention interval. 


This benefit of long inter-study intervals for long retention intervals is apparent in numerous quasi- 
experimental studies of spaced practice in education. Carpenter et al. (2009) conducted learning 
sessions during primary school class time. Eighth-grade children in the U.S.A. were given a history 
test, then a review of answers and facts. This review was then repeated either one week or 16 weeks 
later, followed by a retention test nine months after the review. Children with the ISI of 16 weeks 
recalled more facts after nine months than the one-week ISI group. Bird (2011) compared !SlIs of 
three and 14 days for spaced learning lessons of grammar for English as a second language to 
university students. After a retention interval of seven days, no difference between the two ISIs was 
found, but for a 60-day retention interval, only the 14-day ISI group showed sustained improvement in 
comparison with a pre-test. The three-day ISI group actually significantly decreased in performance 
between the seven-day test and the 60-day test, whereas the 14-day ISI group showed a score 
consistent with their seven-day score. Psychology undergraduates in Canada taught using a spaced 
protocol with eight-day ISIs showed better retention after five weeks than when one-day ISIS were 
used (Kapler, Weston and Wiseheart, 2015). The eight-day ISI group did not differ from the one-day 
ISI group on an initial test that occurred before the learning period, yet the eight-day ISI group scored 
significantly higher than the one-day ISI group on the final retention test, with the eight-day ISI group 
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scoring 15% and 20% more highly than the one-day group on both simple and complex question 
types respectively. 


There has also been an attempt to apply the evidence of a spacing effect on short ISIs from the 
neuroscience literature. In 2013, Kelley and Whatson successfully employed spaced learning 
strategies in the classroom with children aged 14 to 15 years in England, in a quasi-experimental 
study. Teachers were trained to use spaced learning techniques, and assessment was a high stakes 
test on a national curriculum biology course. They claim that 90 minutes of spaced learning (three 
periods of 20 minutes of teaching, interspersed with ten-minute intervals of distractor activities) 
produced retention of the information that was not significantly different to four months of typical 
teaching (massed learning), despite significantly less teaching time. This clearly demonstrates how a 
spaced approach can be considerably more efficient than a massed approach. Kelley and Whatson 
(2013) designed their spacing protocol to mirror the neuroscience work of Fields (2009) who used ten- 
minute ISIs to successfully produce increased long-term potentiation in rat brains. 


Although this is a good example of neuroscientific evidence-based design for educational practice, it 
should be noted that the scientific evidence, from both neuroscience and cognitive psychology, is now 
pointing towards intervals of longer than ten minutes being more beneficial (Kramar et al., 2012), and 
especially so when the retention interval is longer than a few hours (Cepeda et al., 2009; Miles, 2014). 


Interpretation of the literature 


Research in an educational setting (Kelley and Whatson, 2013) suggested that ten-minute ISIs were 
of benefit, as did the rat hippocampal stimulation work by Fields (2009). Systematic reviews (detailed 
in Appendix 1) suggest that longer ISIs are needed for more practical education benefits. Cepeda et 
al. (2006) found that for retention intervals of a month or longer, an ISI of at least one day is needed. 
Later rat hippocampal stimulation research—similar to the work of Fields (Kramar et al. 2012)— 
suggests that 60 minutes is optimal for long-term potentiation, rather than shorter intervals of ten 
minutes. Longer ISIs have been found to be of benefit when long retention intervals with complex 
material are used (Miles, 2014), even in a classroom setting (Kapler et a/., 2015). 


Significant benefits for spacing of synaptic stimulation are similar to behavioural learning (Fields, 
2009; Kramar et al., 2012). However, Kornmeier and Sosic-Vasic (2012) comment that we do not 
know if this long-term potentiation facilitation seen in the lab-based animal studies actually contributes 
to the complex pattern of behavioural benefits (for example, non-monotonic lag effects described by 
Cepeda et al., 2006). This means that we do not know the extent to which these changes in brain 
cells seen in the lab experiments are actually contributing to the real-world changes in memory we 
see in more practical spaced learning studies. For example, these changes in long-term potentiation 
seen in the lab do not offer much explanation of the mechanisms at play when we see that memory 
retention increases up to a certain point when spacing time is increased, but then begins to tail off. 


The vast majority of neuroscience work on spacing effects is very far removed from the classroom. 
EEG research could bridge the gap as it involves the recording of brain activity both during the 
presentation of stimuli and during recall, but there is a need for more focus on the efficacy of longer 
inter-study intervals to determine optimal conditions for meaningful retention periods. Non-uniform 
(Zhang et al., 2011) or adaptive protocols (Kerfoot, 2010) for spacing may be optimal on the individual 
level, but less applicable in classrooms. Non-uniform spacing protocols involve a range of different 
duration spaces within a participant's learning experience, and adaptive protocols tailor the number of 
repetitions to the individual, dependent on performance. These types of spacing may offer 
performance or time benefits over regular spaces and predetermined repetition numbers, however 
they would likely not be feasible for classroom delivery due to the high demand of assessing each 
participant's learning throughout the process. 
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Many hypotheses have been provided for the causal driver of the spacing effect—from protein 
synthesis, electrical activity, through to sleep (a more detailed summary of the research literature is 
available in Appendix 1 and from Gerbier and Toppino, 2015). The cognitive psychology and 
educational practice literature tells us about the mechanisms and benefits of longer spaces. The 
neuroscience literature is focused on shorter spaces. Yet it is important to consider that cognitive 
changes are still at play with shorter spaces, and neurological changes are still at play with longer 
spaces—these pairings have just not been the focus of the respective bodies of literature. However, 
the focus of the current report is to explore the more practical questions around optimum spacing 
length and feasibility in the classroom. 


Overall, we can see that there is evidence in both the neuroscientific and cognitive psychology areas 
of research for a robust spacing effect. However, due to recent progress in both the experimental 
research of cognitive psychology and research exploring spaced learning in an educational context, 
we can see that there are two distinct sources of evidence for short ISIs (from neuroscientific 
literature) and for longer length retention intervals (from cognitive experimental literature). In addition, 
there is limited evidence about the feasibility of using spaced learning in the classroom in both 
literatures and none comparing the two lengths of spaces. In fact, Dempster (1988) has called for 
more applied research on the duration and type of spaces which may be beneficial to learning. 


With this evidence from the research literature in mind, the research and project team co-designed a 
logic model to compare different ISI periods (ten minutes versus 24 hours). Furthermore, we set up a 
further two phases to this investigation. The first phase aimed to test the feasibility of several variants 
of spaced learning with different ISIs in the classroom. The second phase was a small pilot controlled 
trial with the purpose of identifying the optimum variant of spaced learning for improving attainment, 
which would ultimately characterise the SMART Spaces programme. 


Logic model and research rationale 


This section outlines the logic model underpinning SMART Spaces and the rationale for the research 
design utilised in this study (see Figure 1). As an innovation project, this study had three stages 
exploring all the elements of a standard programme design logic model—(1) theoretical development 
(exploring programme inputs), (2) feasibility (investigating programme outputs), and (3) optimisation 
(testing programme outcomes). Theoretical development was conducted using the research literature 
and consultation between the evaluation and project teams. The feasibility study (FS) examined the 
implementation of spaced learning in four schools and trialled a potential science outcome measure 
(GCSE attainment). The optimisation study (OS) was an experiment with a pre-test and post-test of 
GCSE science questions (using the bespoke measure selected from GCSE questions that was 
piloted in the FS) in 12 schools. Three variants of the programme were tested in the OS, with the 
addition of two control groups in order to gain some early insight into the relative efficacy of the 
programme variants. In addition to the outcome assessment, the OS also included a reliable measure 
of pupil engagement (developed by the evaluation team) to investigate the influence of 
implementation on outcomes. 


The study was designed in this way to provide iterative feedback to improve the innovation and 
design of the programme as well as to inform the research design at each stage and produce some 
indication of the promise of the programme improving science attainment. There is one important 
caveat to be remembered about this investigation: this study was NOT set up to be a full efficacy trial 
of SMART Spaces (that is, having an appropriately powered sample size and requisite number of 
schools); rather the purpose was to develop a spaced learning programme, to provide evidence for 
the feasibility of the programme and to optimise it for a future study of its efficacy. 


The SMART Spaces logic model was generated from the evidence provided by the different bodies of 
literature on spaced learning, namely neuroscience and cognitive psychology literature. This resulted 
in the creation of several variants of spaced learning instruction to be tested, each variant mapping 
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onto a theoretical framework provided by the literature. These variants were all designed to be taught 
within a one-hour lesson per day over three consecutive days, with the same amount of time spent on 
teaching the subject material. The lessons were designed to be an hour long so that they would fit the 
majority of school timetables. The only difference between the variants was the length and distribution 
of spaces. These variants were tested in the context of revision lessons for GCSE science, not for the 
initial learning of the science content. The following paragraphs detail the rationale behind each of the 
variants and controls that appear in the logic model. 


Variant 1 (ten-minute variant) 


The neuroscientific literature provides evidence of increased memory markers when ten-minute or 
similarly short spaces are used between bursts of stimulation, in comparison with massed stimulation 
(for example, Fields et al., 2009; Menzel et a/., 2001). This is explored through Variant 1 (called the 
‘ten-minute variant’). In practice, the pupils receive 12 minutes of ‘Subject A’ (for example, chemistry), 
followed by ten minutes of space activity (typically juggling), a further 12 minutes repeating Subject A 
material (the same slides as before), another ten-minute space activity, then a final 12 minutes of 
Subject A. This exact pattern is repeated on day two but with a change to the material provided 
(Subject B, such as physics) and repeated again on day three with material changed once more 
(Subject C, such as biology). 


Variant 2 (24-hour variant) 


Experimental cognitive psychology literature provides evidence that longer spaces (24 hours or more) 
may be optimal when long retention periods are required, such as weeks or months (for example, Bell 
et al., 2014; Carpenter et al., 2009; Cepeda et al., 2009). This is partially explored, with medium-term 
spaces, through Variant 2 (called the ‘24 hour variant’). In practice, the pupils receive 12 minutes of 
Subject A, followed by 12 minutes of Subject B, then 12 minutes of Subject C followed by 20 minutes 
of space activity (to ensure a consistent pattern of 36 minutes content is presented with all variants). 
The same pattern is repeated on day two and day three. The interval may not be exactly 24 hours— 
simply the next day. 


Variant 3 (ten-minute and 24-hour variant) 


Both of these broad approaches (‘ten-minute spaces’ and ‘24-hour spaces’) may contribute to the 
short-term outcome of memory formation and subsequently to the later outcome of educational 
attainment through both similar and different mechanisms of change. To explore this, the research 
was designed to test a third variant, Variant 3, which combined both ‘ten-minute’ and ’24-hour’ spaces 
in the delivery of materials. In practice, the pupils receive 12 minutes of Subject A, followed by 10 
minutes of space activity, then 12 minutes of Subject B, another 10 minutes of space activity, and 
finally 12 minutes of Subject C. The exact same pattern is repeated on days two and three. 


Control 1 (slides only control) 


There are also two controls in this study. Control 1 is a ‘slides only’ control, which provides the 
PowerPoint slides of teaching content in the normal time, with the inter-study task (for example, 
juggling) performed after all content is provided, and thus providing no spaces in the learning. This is 
to assess for any effect that the condensed study slides may have on the learners’ attainment 
outcomes. In practice, the pupils receive 12 minutes of Subject A, followed by 12 minutes of Subject 
A, then 12 minutes of Subject A, followed by 20 minutes of space activity (that is, both ten-minute 
spaces combined at the end). The same pattern is repeated on day two changing the content to 
Subject B, and day three changing the content to Subject C. 


Control 2 (no spaces/materials control) 
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The second control is a pure control, which provided no slides or spaced learning to the pupils and 
they proceed with their normal science lessons. However, they were pre- and post-tested like all the 
other pupils engaged in the variants and control conditions described above. 


In summary, it is important to consider that there is evidence in the literature for both ten-minute and 
24-hour spaces being valuable for memory formation. This study will therefore examine the 
contribution of each of these ISIs, as well as a combination of the two, on science attainment. 


As has been discussed, there are a number of theories of change behind the spacing effect, many of 
which have been explored in the literature (Gerbier and Toppino, 2015). However, it is suggested that 
any observed cognitive learning and neurological changes are not specific to long and short spaces 
respectively. Neurological changes might still occur as a result of longer spacing, they just have not 
been explored in the neuroscience literature. Similarly, cognitive changes might still occur as a result 
of shorter spacing, they are just not the focus in the cognitive literature. The logic model (Figure 1) 
does highlight these links (dashed line) between cognitive and neuroscientific changes as a result of 
spacing effects. Regarding the theory behind the SMART Spaces programme, this study does not 
explicate a theory of change, rather it explores the more practical theory of intervention behind 
spaced learning, that is, on what length of spaces are optimal for producing positive effects on 
memory and learning, and whether it can be delivered in a feasible way in a classroom. 


Lastly, it is well understood that educational programmes do not operate in a fully controlled 
environment. Therefore, it is important to understand contextual factors and their influence on 
programme effectiveness (Bonell, et a/., 2012; Craig, et al, 2008; Jamal, et al., 2015). To do this we 
explored the influence of implementation factors on programme effects. One major issue explored 
was pupils’ engagement with the spaced learning—using a standardised post-test measure—as well 
as the type of spaced activity (was it juggling or some other activity?) so we could relate these factors 
to any observed outcome change. 


Research questions 
With the logic model and rationale in mind, we considered the following research questions: 


1. What can we learn about the feasibility of implementing a ‘spaced learning’ programme ina 
practical classroom situation? 

2. What length of spaces showed the best evidence of promise on attainment outcomes? Was it 

ten-minute spaces, 24-hour spaces, or a combination of the two? 

What implementation factors have an influence on programme effects? 

4. Is SMART Spaces feasible enough, and with enough evidence of promise, to warrant a future 
efficacy trial? 


w 
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Figure 1: SMART Spaces Logic Model and Research Rationale 
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Ethical review 


All research was conducted according to QUB School of Education ethical guidelines. Ethical consent 
was obtained from the Ethics Committee before data collection was conducted. Informed consent was 
obtained from participant schools and all pupils had a right not to complete or withdraw their data. The 
data collected was coded and entered onto a database, made anonymous, and held securely on a 
password-protected computer. No NPD data was required for this project. 


Project team 
Centre for Evidence and Social Innovation—Evaluation Team 


e Dr Liam O’Hare (Principal Investigator) 
e Dr Andy Biggart 

e Professor Carol McGuinness 

e = Dr Patrick Stark 

e Professor Allen Thurston 


Hallam Teaching School Alliance—Project Team 


e Alastair Gittner (Project Team Leader) 
e Farhana Zaman (Project Manager and Administrator) 
e Steve Davis (Head of Hallam TSA) 
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Methods 


Recruitment 


The schools were recruited in both study phases (feasibility study and optimisation study) by the 
project team." Four schools were recruited for the FS, and 12 for the OS. The four schools in the FS 
had already been using a version of spaced learning (predominately a ten-minute variant) for a few 
years and had a range of familiarity with the concept. The 12 schools in the OS had not previously 
implemented spaced learning. For these schools, recruitment advertisements were shared across 
England by the implementation team who then responded to interested schools with further 
information. Only 12 schools agreed to take part. There were no selection criteria for the 12 schools in 
this part of the study beyond an expression of interest, and none were excluded from the study. 
However, all schools were in the Yorkshire and Lancashire area of England and most had a high 
percentage of pupils in receipt of free school meals (a proxy measure for disadvantage). 


Consent was sought at the school level through a memorandum of understanding (see Appendix 3). 
The 12 schools in the OS were randomised by the evaluation team into the five conditions: 


e Variant 1 (ten-minute version): three schools; 

e =Variant 2 (24-hour version): two schools; 

e Variant 3 (combined ten-minute and 24-hour version): three schools; 
e Control 1 (‘slides-only’ control): two schools; and 

e Control 2 (‘business as usual’ control): two schools. 


All GCSE science pupils in the school were eligible for the programme on the condition that their class 
teachers had received the SMART Spaces training. Classes of pupils were chosen within each 
participating school by the Spaced Learning contact for each school. The teacher who returned an 
expression of interest may have volunteered their own teacher time, and may have asked other 
teachers to also participate in the study. Schools did not include all GCSE science pupils—one to two 
classes were chosen in each school by the participating teachers. Consent was sought at the pupil 
level through opt-out consent forms (Sent home to parents and verbally explained to pupils at testing) 
for participation in the pre-tests and post-tests. 


Data collection 
1. Theoretical development (months 1-6) 


In order to gain insight into potential theories to underpin the SMART Spaces programme, a 
structured literature review of ‘spaced learning’ interventions was conducted covering the elements of 
a standard logic model—inputs (resources), outputs (activities) and outcomes (changes), 
assumptions (relating to training, quality of delivery, engagement, and so on), and external factors 
(current ‘spaced learning’ practice and proliferation as well as the educational context). The literature 
review also examined the underpinning theory around spaced learning in terms of a theory of change 
and a theory of intervention. This literature review covered multiple perspectives on spaced learning 
research and included neuroscientific, cognitive psychological, and educational practice literature. 


The literature review fed into a service design and logic modelling process. This process was a 
collaboration between a range of stakeholders: the evaluation team (QUB), project team (Hallam 
TSA), school principals, and teachers. The process began with a programme development retreat that 


* Recruitment for the OS was the responsibility of the project team, with support and assistance from the 
evaluators. All schools recruited were English State schools. Schools had varying degrees of familiarity with the 
concept of spaced learning but none was implementing it. 
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included capacity-building in logic modelling, presentations on spaced learning literature, and current 
spaced learning practice. 


The logic model produced during the retreat informed the design of the spaced learning programme 
(see Figure 1). Three variants of spaced learning and two control conditions were designed as a 
result of this theoretical development process. The evidence from the early neuroscience literature 
informed the development of the ten-minute space condition. Evidence from cognitive psychology, 
more recent neuroscience literature, and educational practice was the basis of the 24-hour space 
condition. A combined approach of using both ten-minute and 24-hour spaces was also designed 
(with the assumption it could bring together the impact of both short-term and medium-term 
mechanisms of change). A control condition, using the spaced learning teaching materials but without 
the spaces between learning periods, was designed to test for the effect of the materials alone. This 
design was explored with a group of teachers and the project team to develop activities, PowerPoint 
resources, training, and lesson plans for the FS pilot study. 


2. Feasibility study (months 7-12) 


Having produced a draft manual in the first phase, the feasibility study explored the implementation of 
the programme (with its three variants and one control) in practice. The project team worked closely 
with a small number of schools in this study (n = 4).? Classes in each participating school were 
assigned to the three different variants of spaced learning or to the control condition. Eight classes 
participated in this study: four classes were assigned to the ten-minute variant, one to the 24-hour 
variant, two to the combined ten-minute and 24-hour variant, and one to the ‘no spaces’ control 
condition. All schools had one class implementing the ten-minute variant—as they were familiar with 
its use—but also agreed to try one of the other variants as well (See Table 4). 


Table 4: Four feasibility study schools and their variants piloted 


Tor aTele) | Class 1 Variant Class 2 Variant 
1 ten-minute ten-minute/24-hour 
2 ten-minute 24-hour 
3 ten-minute no spaces control* 
4 ten-minute ten-minute/24-hour 


¥i.e. using the spaced learning teaching materials but without the spaces between learning periods. 


To initiate the FS, the teachers met with the evaluation and project teams. The group of teachers 
implementing the programme in the FS had familiarity with the ten-minute programme and agreed to 
deliver it or train their colleagues when required to provide the programme. No teachers were trained 
in the other programme variants before the FS was conducted. However, it was explained to them 
how to change their processes to deliver the other variants during the meeting. 


Five pupils in each school took part in focus groups (20 pupils in total across the four schools) to 
explore engagement, enjoyment, and classroom practicality of each variant. One teacher or 
headteacher from each school (n = 4) was interviewed at this stage to investigate engagement with 
the research project, implementation of the lessons, training, and support procedures. 


In addition, after implementing the spaced learning programme, pupils took part in a pilot of the 
outcome measure that was to be used in the next study to check its usability. This test was developed 
and administered by the evaluation team. This was to explore the appropriateness of the bespoke 
science measure, rather than to examine quantitative differences between variants at this early stage. 


? This was an increase from the three schools stated in the protocol because there were four schools engaged 
and interested at this stage. 
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3. Optimisation study (months 13-17) 


The next phase was the design of the optimisation study involving 12 schools. Schools were 
randomised at the school level into one of the three intervention conditions or one of the two control 
conditions—these were the four conditions studied in the FS plus an additional ‘business as usual’ 
control (See page 18). Feedback from the FS highlighted the need for teachers who were delivering 
the programme to be well trained, and thus all teachers received training on spaced learning, 
including additional practice and feedback on the specific variant that they were being asked to 
implement, as well as being provided with the appropriate materials. Pupil data was collected 
quantitatively through the pre-test and post-test science attainment outcome measure piloted in the 
FS. The test was administered by the evaluation team (the evaluation team were aware of the variant 
the school was testing, but the project team did not know the content of the test). The pupils also 
received a post-test implementation questionnaire to investigate implementation, fidelity, and pupil 
engagement with the programme (see Appendix 2). One teacher from each school who delivered the 
lessons from each variant (total n = 4) was also interviewed to explore implementation factors such as 
fidelity, training, and sustainability. 


4. Review and final manual production (months 18-21) 


The final phase of the project was ongoing with the qualitative and quantitative findings feeding back 
into the logic modelling process (with the aim of revising it to produce the final programme logic 
model—see discussion) and the production of this report. Final refinements are being made to the 
programme design in a series of meetings between the evaluation and project teams. These 
discussions will produce the final output in the form of a ‘spaced learning’ manual featuring a literature 
review, logic model, implementation data, a training guide, and programme content. This manual will 
be available for use in any future effectiveness study of the intervention. 


Randomisation 


For the OS, randomisation was conducted at the school level. Schools were ordered in terms of 
numbers of participants and then divided into two groups based on participant numbers (Group A = 
six schools with largest participant numbers, Group B = six schools with smallest participant 
numbers). Random numbers were generated for each group to allocate them to one of the five 
conditions. The remaining schools, one in each group, were allocated to the ten-minute and 24-hour 
variants respectively (to ensure some numbers in these two variants in case of school withdrawal). 
Four schools were pre-tested after randomisation. Randomisation took place on 23 February 2016; 
four schools were pre-tested in a window two weeks after that, up to 8 March 2016. 


One school (School L, a small independent school) assigned to the 24-hour space variant did not 
wish to take part in the training and delivery of the programme but agreed to the pre-and post-test 
(with 11 of the 14 pupils providing complete data). Therefore, it was reassigned to the ‘no 
spaces/materials’ control group. The study cannot, therefore, claim to be a fully randomised trial. 
However, as design, feasibility, and optimisation were the foci of the study showing evidence of 
programme promise rather than actual efficacy of the programme, it was deemed appropriate by the 
evaluation team to include this school in the analysis. 


Measures 


Outcome measures were conducted after implementation of the programme in the FS, and at both 
pre-implementation and post-implementation time points in the OS. The same outcome measure was 
used on all three occasions—a bespoke science measure, comprising past-paper questions from the 
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AQA GCSE® curriculum. This test comprised two sections: Section A—short answers and multiple 
choice, and Section B—extended answers. There were 39 marks available for Section A and 18 
marks for Section B—a total maximum score of 57. Appendix 4 gives full details of the GCSE topics 
for each science subject (biology, chemistry, and physics) for each section in the paper, the marks 
allocated to each topic, and the source GCSE science paper. The short answer and multiple choice 
section required participants to give answers ranging from one word to two or three lines; the 
extended answer section required considerably more detail per answer, requiring five to six key points 
of information. The test had a time limit of 45 minutes in the optimisation study, which was determined 
by the average time taken to complete the test in the feasibility study. The project team were blind to 
the content of the outcome measure so as not to influence the content of the training sessions or 
slides to include additional emphasis on the exam questions used. The reliability of the six scores 
(Sections A and B for the three subjects) used to calculate the total score on the test had a 
Cronbach’s alpha = 0.88. Pre-tests occurred between 23 February and 13 April 2016; post-tests 
between 26 April and 20 May 2016. Typically, the period between pre- and post-tests was 62 to 65 
days with the exception of two schools where it was 49 and 37 days. The period between pre-test and 
post-test for each school is shown in Appendix 6. Gain scores (post-test minus pre-test scores) were 
used in the analysis of the data collected and these are summarised in Appendix 8. 


Data from the focus groups provided during the FS was used to design an implementation 
questionnaire which was administered to pupils post-implementation in the OS (see Appendix 2 for 
summary statistics of responses). This questionnaire also included a ‘pupil engagement in spaced 
learning’ scale which was used in outcome-related analysis (see Table 9). There were 13 items in this 
engagement scale (see Appendix 2 for more details). Reliability of this measure was very good 
(Cronbach’s alpha = 0.91). 


Sample size* 


As this project was largely a design, feasibility, and optimisation process, sample size was not based 
on required numbers to identify effectiveness. Rather, sample size was chosen to best suit the 
feasibility and optimisation testing process. Four schools initially implemented a draft version of the 
programme and its variants. Twelve schools participated in the OS stage of the research. Table 5 
below shows the number of participant pupils in each of the 12 schools, broken down into the variants 
that they received. 


3 AQA is a U.K. exam board, and GCSEs (‘General Certificates of Secondary Education’) are national subject- 
specific awards typically taught and conducted in the U.K. in Years 10 and 11. 

No power calculation was conducted in this project because it was a design, feasibility, and optimisation study 
of a programme and not a trial of efficacy. A power calculation would have suggested the recruitment of many 
more schools than actually participated in this study in order to produce findings with a high degree of 
power/security in terms of programme efficacy. The findings in this study are very low in efficacy security but very 
useful in terms of design, feasibility, and optimisation. 
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Table 5: Variant by school and pupil numbers 


WENaEal! Pupil numbers in schools A-L 
B G 

10-min 0 | 42 0 0 0 0 0 0 | 47 | 21 0 0 110 
24-hr 0 0 0 0 0 0 0 | 44 0 QO | 31 0 75 
10-min and 24-hr 0 QO | 21 0 | 43 | 27 0 0 0 0 0 0 91 
no spaces control 39 0 0 0 0 0 | 40 0 0 0 0 0 79 
no spaces or 0 0 0 | 42 0 0 0 0 0 6) Oo | 11* 53 
materials control 

Total 39 | 42 | 21 42 | 43 | 27 | 40 44 | 47 | 21 | 381 11 408 


*Reassigned from 24-hour version. 


Analysis 


Analysis was conducted after each of the two stages, the FS and the OS. After the FS, the pupil focus 
groups and interviews fed into the logic modelling and research design retreat at the end of the FS. 
Furthermore, reliability and validity tests of the bespoke measures were conducted to explore whether 
the measures were valid, acceptable, intelligible, collectable, and capable of showing plausible 
positive changes. 


At the OS phase there was an analysis of the effects of each variant on pre-post outcome measures 
compared to both control groups. These were calculated using mean gain scores, pre and post SDs, 
and paired t-tests. Variant scores were compared to control scores on total score short-answer and 
long-answer questions. 

The analysis also explored the relationship between implementation factors (engagement, space 
activity type, and retention) and outcome change. No clustering was accounted for in these regression 
analyses as there were five arms to the optimisation experiment and only two or three schools in the 
five arms. However, multilevel models were conducted and are included in Appendix 7 to provide the 
reader with this information if required. 


Costs 


The following cost estimates were developed during the creation of the SMART Spaces manual: 


Cost item Per class (£) 
Cost of one class manual 100 
Cost of juggling balls for one class 10 
Cost of trainers time/number of teachers at training (£450/5) 90 
Cost per class in first year 200 
Cost per pupil (20 pupils per class) in first year 10 


Trainers were paid on a per-hour basis, (around £50 per hour + travel expenses with an 
understanding that preparation time was needed for each session). 


Teachers are required to attend one full day of training, and cover will need to be arranged for this. 
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Timeline 


Table 6: Timeline of project activities 


Activity 


e Ethics application QUB School of Education 
e Collect information for literature review 

e §=© Start recruitment of four initial FS schools 
e 

e 


Finalise schools Memorandum of Understanding 
Plan and book retreat to develop programme and research 
protocol 


Feb Theoretical Structured literature review (around elements of a logic model) 
development looking at spaced learning from both a neuroscientific and 
cognitive psychology perspective 
Mar/Apr/May Design initial logic model(s) 


Draft program development retreat agenda 

Capacity building on logic modelling (QUB) 

Literature review presentation (QUB) 

Discuss ‘pure control’ 

Recruit 12 schools for OS trial 
Feasibility Group of teachers worked with the evaluation and project teams 
study (FS) to develop activities, PowerPoints, training, and lesson plans 

Draft training manual 
July/Aug/Sep Reflect on the quality of the materials, whether or not they are 

useful and engaging, etc. 

Review manual including: literature review, logic model, training 
manual, and delivery materials 
FS conducted with 4 schools 
e Developed bespoke science test based on past GCSE science 
papers 


Developed implementation survey 


e 

e Analyse data 

e Review logic models and materials based on pilot 

e Pre-test in 12 OS schools with bespoke science test 


e =§=Training and implementation of programme 
e =§=Training and implementation of programme 
7 - : : : 


Post-test measures, bespoke science test an implementation 
measure 


e Analyse OS data 


Review and|e_ Present final data 
finalise 


July e Revise programme manual (including literature review, logic 
model, training manual, and delivery materials) 
e Programme manual graphic design 


Aug/Sep e Finalise research report and submit to the EEF 
e Finalise SMART Spaces manual and submit to the EEF 


Optimisation 
stud 
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Findings 
Feasibility study 


This section summarises the key findings from the FS, which evaluates whether the intervention is 
likely to be practical and feasible. In essence, it answers the first research question: 


What can we learn about the feasibility of implementing a ‘spaced learning’ programme ina 
practical classroom situation? 


In order to do this, interviews were conducted with eight teachers and headteachers who had a range 
of previous experience with spaced learning. Focus groups were also conducted with five children in 
each of the four participating schools in the FS (20 children in total). The analysis of qualitative data 
and subsequent programme adaptations were based on emergent themes rather than on any one 
individual’s comments. 


Outline of qualitative results from the feasibility study 


We received substantial positive feedback from both teachers and pupils about the programme. The 
qualitative data provided insight into the benefits and limitations of spaced learning, as seen by pupils 
and teachers. As we progressed towards the OS stage, the interviews and workshops highlighted 
some specific areas for adaptation to the programme delivery, materials, and training. 


e Programme adaptations 

o Teachers reported the importance of needing to receive the resources well in 
advance, as, in numerous instances, materials arrived too close to planned delivery 
days, or in one instance, the juggling balls arrived a day late. 

o Teachers and headteachers reported that last minute changes-of-plan did not 
consider the busy school environment. Any school visits or scheduling changes must 
be planned as far in advance as possible. 

o Juggling: one teacher reported that children who struggled to learn this skill ‘gave up’ 
and were frustrated or bored by the third lesson. Teachers suggested other distractor 
activities such as balloon games, transformer toys, and origami (all used successfully 
before). Balls were reportedly of poor quality. ° 


ACTION for the OS: Timely delivery of materials and resources are needed to reduce pressure on 
staff. Consideration should be given to expanding the training relating to juggling to help teachers 
maintain pupil engagement with both the activity and the lesson. 


e Material adaptations 

0 The length of the PowerPoint (PPT) presentations was inconsistent between topics 
and subjects, according to both children and teachers. Many teachers felt that the 
presentations were too long, and it was a difficult to fit them, and juggling, into an 
hour’s lesson. 

oO Several teachers suggested that the materials would benefit from more proof-reading, 
and that some of the science content should be double-checked and updated to 
ensure it meets the most recent curriculum specifications. 

o There were some problems with visuals on the biology slides, for example, some of 
the pictures were covered over. 


? Analysis of the OS would not suggest that using juggling balls has a deleterious influence on effects, so juggling 
will be still be recommended in the SMART Spaces manual as the space distractor task. However, instruction will 
be provided in SMART Training to ensure pupil engagement in the task. 
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o The synchronisation of slides was sometimes off. For example, a missing word would 
appear on a slide, children would shout the answer, but the correct answer would not 
appear and the PPT skipped straight to a new gap. Teachers reported having to 
modify the PPT structure to ensure it ran smoothly. 

o ‘Each PPT was a different style...varied in length, varied in animations...it needs to 
be a consistent design’, reported one teacher. 

o Pupils and teachers reported that the layered style of the slides seemed ‘old 
fashioned’. 


Action for the OS: PPTs need to be checked to ensure that al/ variants fit within one hour, including 
breaks. They also require extra proof reading. The flow and synchronisation of slides needs to be 
checked, and similar styles between subjects will give a more consistent experience. 


e Training adaptations 

o Teachers reported that they should have been advised not to have three different 
teachers running the three different portions of the ‘24-hour’ lessons. When this 
happened it resulted in organisational problems, made the lessons harder to follow, 
and the children felt they were rushed and less engaged. 

oO Some teachers reported that the set-up meeting between teachers, evaluation team, 
and project team did not provide sufficient training for this project.° Those with prior 
experience of spaced learning were not comfortable with other variants of the 
lessons. Further modelling and practice of the different programme variants could 
have been provided. 

o Pupils noted that different teachers showed different levels of comfort with delivery. 
This may be reflective of the fact that not all teachers were directly trained. 

o Slides in training were different from the slides used during delivery. Teachers 
expressed a need for training using the specific slides they would be using. One 
teacher felt that there was an assumption that teachers would know the ‘story’ of the 
presentation, and how things were connected, when this was not readily apparent. 

Oo One teacher reported not having had training specific to the project. This made the 
24-hour space version very difficult to teach. They had only been trained on the ten- 
minute space version. 

o Numerous teachers suggested that getting a chance to practice spaced learning, with 
feedback, would be very useful—perhaps more examples of ‘good’ spaced learning 
practice versus ‘bad’ practice. Teachers felt there should be a ‘standardised model’ of 
the spaced learning variants to refer to. 

o Teachers who had to train other members of staff within their department felt this was 
not adequate. 

o Children reported that some teachers tried to add in too much extra information to the 
slides. This could be addressed in training. 


Action for the OS: All teachers delivering the programme need to be trained. Training needs to 
specify that one teacher will deliver all lessons within a variant, even if the topic is not their specialist 
subject. Teachers need to be given more confidence in delivering a ‘good’ spaced learning lesson, 
and need to come away knowing what a standardised spaced lesson looks like. Training should use 
the exact materials the teachers will be delivering. All these concerns are now addressed in the 
SMART Training session and SMART Spaces manual. 


e Feedback on specific variants 


® Teachers in the FS were only given information about the programme and delivery at the programme 
development retreat. SMART Training was developed and implemented at the OS stage. 
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o Variant 1 (ten-minute spaces). Teachers felt most comfortable delivering this as 
many had prior experience of spaced learning in this style. The children who received 
these lessons reported that their teachers seemed confident and enthusiastic—more 
so than children receiving other variants. 

oO Variant 2 (24-hour spaces). Children and teachers felt this variant was more rushed 
and difficult to deliver as some schools attempted to use three different teachers for 
the three different subjects. The concept of 24-hour spaces between lessons seems 
to have been less noticeable to participants than the challenge of squeezing three 
teachers into one lesson. The lack of training, and lack of prior experience of this 
variant, made teachers less confident. 

o Variant 3 (ten-minute and 24-hour combination). Children reported enjoying the 
juggling breaks and feeling engaged, but teachers reported that this variant was too 
long and too difficult to fit into one hour. Ensuring that only one teacher delivers this 
lesson should help with timing issues, but content editing for all variants seems 
necessary due to the frequent reports of difficulty fitting into one hour. One teacher 
reported finding this variant tiring to deliver. 


Action for the OS: It was judged that although there were difficulties in delivering specific variants 
these could be overcome by the adaptations to the programme, materials, and training as described 
above. Furthermore, it was deemed necessary to optimise the programme through a preliminary 
check of the efficacy of the different variants. 


Performance of measures 


One hundred and sixty-nine pupils completed the bespoke GCSE science assessment at post-test in 
the FS. Reliability tests showed a Cronbach’s alpha score of 0.85. Furthermore, total scores on the 
test were found to be normally distributed thus indicating no floor or ceiling effects of the test. In 
addition, no pupil scored O or the maximum of 57. 


Optimisation study 

Participants 

Twelve schools participated in the optimisation study phase. Table 7 shows five key characteristics for 
each school: (1) whether urban or rural, (2) the size of the school, (3) the proportion of pupils in 


receipt of free school meals (‘FSM pupils’—an indicator of level of deprivation), (4) the type of school, 
and (5) its Ofsted rating. 
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Table 7: The characteristics of the 12 schools involved in optimisation study 


sSYoiaTeye)| Variant Type Urban/ Si74-m0) | FSM Type of 
Rural SYouareye)| Pupils (%) Establishment 
A No spaces Urban Academy Good 
1755 15 Converter 
B 10-min Rural 1522 3 Academy Outstanding 
Converter 
Cc 10-min & 24-hr Urban 725 15 Community Outstanding 
School 
D No Urban 525 21 Academy Good 
spaces/materials Converter 
E 10-min & 24-hr Urban 1650 15 Academy Good 
Converter 
F 10-min & 24-hr Urban 980 21 Academy Good 
Converter 
G No spaces Urban 1244 15 Academy Good 
Converter 
H 24-hr Urban 1600 10 Community Good 
School 
| 10-min Rural 600 16 Academy Good 
Converter 
J 10-min Urban 600 12 University Good 
Technical 
College 
K 24-hr Urban 1365 35 Academy Good 
Sponsor Led 
L No Urban 160 0 Other N/A 
spaces/materials Independent 
School 


Evidence to support theory of change 
Comparison of variant effects 


To get an indication of effects of the programme variants over the different groups, we compared the 
effect sizes (ES = Hedge’s g using mean gain scores, pre and post SD’s, and paired t-tests) of the 
different conditions and compared them to the control (Table 8).’ 


It can be seen that Variant 3 (the combined ten-minute and 24-hour variant) showed a pattern of 
positive effects on all scores and against both controls, with one significant difference showing greater 
positive change between Variant 3 (total score) and the no spaces/materials control (ES = 0.19). 
There was a more modest effect of the Variant 1 (ten-minute version) across the scores and controls 
with no significant effect on any of the comparisons. Variant 2 (24-hour version) produced some low 
and some negative effects across all comparisons with one significant negative effect between the 
Variant 2 long answer score and slide-only control (ES = -0.37). Therefore, it can be seen that Variant 
3 shows the best and most consistent evidence of promise against controls at this stage of 
programme development. 


: Appendix 8 provides a table showing pre, post, and gain score summary (across the five groups). This table 
also includes gain score removing school L which was reallocated to the ‘no spaces/materials’ control group. It 
suggests that school L had better gains than the other school in this control group. Therefore, if anything, school 
L dampened the effect size difference between control and the three variant types. 
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Table 8: Effect sizes for the three variants on total, short, and long answer scores compared 
against the two control groups 


NVEVar-Vol may o1-) (exe) alice) Test TayccdaV{:1alice) ames ar-lole (= Mmmm Oxo) alice) Mesar-lale l= Effect size =g 
group score (pre s.d., post s.d., (pre s.d., post s.d., (95% confidence 
type type N, paired t) N, paired t) intervals) 

10-min Slides Total 2.89 2.62 0.03 
only (9.09, 9.82, (8.54, 8.89, 68, (-0.14, 0.20) 

100, -5.49) -4.39) 

10-min Slides Short 2.26 1.62 0.10 

only (6.25, 6.58, (6.05, 6.60, (-0.13, 0.33) 
100, -5.47) 68, -2.57) 
10-min Slides Long 0.63 1.00 -0.09 
only (4.07, 4.41, 100, (3.56, 3.72, 68, (-0.34, 0.16) 
-2.47) -2.43) 
10-min Nothing Total 2.89 1.82 0.12 
(9.09, 9.82, 100, (7.16, 9.79, 50, (-0.07, 0.30) 
-5.49) -2.48) 
10-min Nothing Short 2.26 1.42 0.13 
(6.25, 6.58, 100, (5.41, 6.94 50, (-0.08, 0.34) 
-5.47) -2.80) 
10-min Nothing Long 0.63 0.40 0.06 
(4.07, 4.41, 100, (2.88, 3.90 50, (-0.21, 0.33) 
-2.47) -0.90) 

24-hr Slides Total 1.54 2.62 -0.09 

only (12.81, 14.63, 69 (8.54, 8.89, 68, (-0.25, 0.07) 
-2.36) -4.39) 

24-hr Slides Short 2.03 1.62 0.05 

only (9.23, 10.81 69, (6.05, 6.60, 68, (-0.17, 0.27 
-3.66) -2.57) 
24-hr Slides Long -0.49 1.00 -0.37* 
only (4.13, 4.62, 69, (3.56, 3.72, 68, (-0.63, -0.11) 
1.65) -2.43) 
24-hr Nothing Total 1.54 1.82 -0.02 
(12.81, 14.63, 69 (7.16, 9.79, 50, (-0.20, 0.14) 
-2.36) -2.48) 
24-hr Nothing Short 2.03 1.42 0.07 
(9.23, 10.81 69, (5.41, 6.94 50, (-0.11, 0.25) 
-3.66) -2.80) 
24-hr Nothing Long -0.49 0.40 -0.22 
(4.13, 4.62, 69, (2.88, 3.90 50, (-0.50, 0.06) 
1.65) -0.90) 

10-min & 24-hr Slides Total 3.76 2.62 0.11 

only (10.59, 12.22, 76 (8.54, 8.89, 68, (-0.05, 0.28) 
-6.55) -4.39) 

10-min & 24-hr | Slides Short 2.53 1.62 0.12 

only (8.10, 8.70, 76, (6.05, 6.60, 68, (-0.11, 0.35) 
-5.65) -2.57) 
10-min & 24-hr Slides Long 1.24 1.00 0.06 
only (3.65, 4.30, 76, (3.56, 3.72, 68, (-0.21, 0.33) 
-3.88) -2.43) 
10-min & 24-hr Nothing Total 3.76 1.82 0.19* 
(10.59, 12.22, 76, - (7.16, 9.79, 50, (0.01, 0.36) 
6.55) -2.48) 
10-min & 24-hr Nothing Short 2.53 1.42 0.15 
(8.10, 8.70, 76, (5.41, 6.94, 50, (-0.03, 0.33) 
-5.65) -2.80) 
10-min & 24-hr Nothing Long 1.24 0.40 0.22 
(3.65, 4.30, 76, (2.88, 3.90, 50, (-.07, 0.51) 
-3.88) -0.90) 
*Significant < 0.05 
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To further demonstrate the pattern of effects, Figure 2 shows an error bar chart (with 95% confidence 
limits) of the mean effect size, comparing the mean variant effect size across both controls. 


For example: 


[(ten-minute total versus ‘slides-only’ control, ES = 0.03) + (ten-minute total versus ‘no 
spaces/materials’ control, ES = 0.12)] divided by 2 = 0 .075. 


Again, Figure 2 shows a clear pattern of higher effect sizes for Variant 3 against the other two 
versions. In fact, it is particularly beneficial on the long answer questions over the other two 
conditions. It is worth noting the particularly poor performance of the ten-minute and 24-hour versions 
on long answer questions, with the 24-hour version showing a large significant negative effect on 
these questions. 


Figure 2: Error bar chart (with confidence intervals) showing a comparison of mean variant 
effect size compared to both controls, for all scores (total, short answer, and long answer) 


pe 10/24 Long Answer 
po 10/24 Short Answer 
| 10/24 Total Score 
[oe 24hr Long Answer 
a 24hr Short Answer 
pe 24hr Total Score 
a 10min Long Answer 
po 10min Short Answer 
p—-- 10min Total Score 


Implementation Factors 


The next step was to examine the relationship between contextual factors and effects of the 
approach. 


Engagement 


One of the important contextual factors reported by the project team was the engagement of teachers 
and pupils in the programme. Therefore, a spaced learning engagement measure was developed and 
administered to the pupils at post-test reporting on their teachers’ and their own engagement in the 
programme. This measure was found to be reliable (Cronbach’s Alpha = 0.91) and was regressed 
onto outcome change (Table 9). Two hundred and twenty-four pupils received one of the three 
variants of the intervention, completed pre- and post-tests, and the engagement questionnaire. On 
analysis of the relationship between their outcomes and engagement it can be seen that this 
engagement score was a significant implementation factor, with higher engagement score predicting 
more positive outcome change (the adjusted R Square for the model was 0.81 showing the high 
degree of the variance in post-test score being predicted by pre-test and engagement score). 
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Table 9: Regression of independent variables pre-test and engagement onto post-test outcome 
scores 


Unstandardized Standardized 
coefficients coefficients 
Std. Error 
1 (Constant) -0.97 2.04 -0.47 0.64 
| Pretesttotal = 099 003 0.40 3054 <01. 


score (max=57) 


Engagement 0.11 0.05 0.06 2.13 0.03 
score 


Dependent Variable: post-test total score (N = 224) across all variant types (max score = 57). 


Inter-study activity 


Another frequently occurring question concerned what activity was best to use in the ten-minute 
‘spaces’—was it juggling or some other activity? Some schools did use activities other than juggling, 
but only a low number (juggling n = 214, other activity n = 18). An independent samples t-test showed 
that those who used juggling had significantly higher outcome change scores (juggling mean = 3.09 
and s.d. = 5.03) compared to using another activity (mean = -0.83 and s.d. = 5.14) (t(df) = 3.17 (230), 
p = <0.01). However, this result must be viewed with caution given the small number who did not use 
juggling and the fact that this could be a proxy for engagement as all teachers were instructed to use 
juggling in the ten-minute spaces. Furthermore, pupils were reporting using an activity other than 
juggling across a number of schools, although it seemed to be one school that had the majority of 
pupils reporting other activity (N = 8). There may have been a mix of use of other activities and 
reporting errors. This result must therefore be treated with caution. 


Retention 


Previous literature has suggested that the length of spaces has an influence on the period of retention 
of information. The final factor investigated was the influence of retention on programme effects. 
There was some variation in the time between the programme variant ending and the post-test 
occurring (due to Easter holidays). Of the 313 pupils who completed pre- and post-tests and received 
a programme variant (this includes the ‘slides-only’ control group, but not the ‘business as usual’ 
control group), there was a mean interval of 18.19 days (s.d. = 12.44) between the end of the 
programme and the test date with a minimum gap of seven days and maximum gap of 48 days 
reported. This variation was exploited to explore the influence of retention interval. To do this, a 
regression was conducted which showed no significant effect of retention interval on the post-test 
scores (see Table 10). 
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Table 10: Regression of retention interval and pre-test score onto post-test score outcome 
variable 


Unstandardized Sl elalel-lgely4-10| 
coefficients coefficients 
Std. error 
1 (Constant) 3.520 0.969 3.632 <0.001 
Pre-test - total 0.977 0.028 0.893 34.440 <0.001 
score (max = 57) 
Time lapsed -0.012 0.025 -0.012 -0.457 0.648 
between 
programme end 
and post test 


Dependent Variable: post-test - total score (N = 313) across all variant types (max score = 57). 


Outline of qualitative results from the optimisation study 


Interviews were conducted with teachers who delivered each of the spaced learning variants (not 
including control conditions). The qualitative data of teachers’ perspectives from this optimisation 
study shows that the actions determined by the feasibility study have been addressed successfully. 


Training and support from the delivery team 


e Teachers reported that the training session was very enjoyable. The small size of training 
groups was appreciated as it facilitated extensive personal feedback and a chance to refine 
lesson delivery. 

e One teacher suggested that the training could have focused more on how to explain to the 
pupils the ‘science behind’ spaced learning to encourage engagement with the project. 

e The face-to-face nature of the training session was perceived as very appropriate, as training 
for this style of teaching, ‘definitely does need to be in person’. Teachers reported receiving 
useful advice, for example, to restrict ad-libbing and stick to materials for the purposes of the 
research. 

e The successful training meant that most teachers did not feel they needed much further 
support to deliver the intervention. 


Materials 


e Teachers reported that the PowerPoint slides were very high quality, and although they didn’t 
always fit exactly with their exam board curriculum, they felt comfortable using the slides in 
the lessons. Others felt the content of the slides was not consistently engaging. 

e A few spelling errors and American spellings were noticed. 

e One teacher suggested that today’s pupils are used to slightly more ‘high tech’ lesson 
materials, and that something more engaging and interactive than PowerPoint slides could be 
used. 

e Versions of the slides for the visually impaired should be available on request. 
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Fidelity of delivery 


e Teachers generally reported that they stuck exactly to the provided materials. No problems 
were reported relating to running the three days, aside from some minor timetable changes. 

e Some adaptations had to be made to PPT slides, such as spelling, definitions that did not fit 
with the children’s knowledge of the curriculum, and some colour/visual changes. 

e One teacher reported switching to ‘paper balls’ to stop children bouncing the juggling balls 
around the classroom, and problems with balls disappearing at the end of class. 


Pupil engagement and wider impacts 


e Many teachers reported that they felt pupils in general engaged well with the lessons. 

e Low ability students were considered more likely to ‘get more out of it’. 

e Higher ability students—A*, A, and top B—were considered more likely to get bored due to 
repetition, or to feel like the material wasn’t sufficient to improve their grade. 

e One teacher reported that his higher set actually engaged better with spaced learning, but 
reported that they ‘didn’t feel like they had learned anything’. 

e Numerous teachers have gone on to continue running spaced learning with other classes and 
subjects after the project. One suggested that they will keep using spaced learning, but not 
with juggling—maybe ‘mindful colouring’. 

e They reported being able to use the layout of the slides to suit their syllabus, and that it was 
great to be able to hold on to them and use them for further spaced learning in the school. 


Feedback on specific variants 


Variant 1 (ten-minute spaces). One teacher reported no major issues with implementing this lesson 
style and that it ran smoothly. Another, however, reported that their lower ability students found the 
pace too difficult. Class control was an issue for one teacher due to the frequent bouts of physical 
activity in the class. 


Variant 2 (24-hour spaces). The first session, which is the most content-heavy in the variant, was 
difficult to run and was longer than later sessions. One teacher reported that students really ‘latched 
on’ to the lessons towards the end of the program, despite engagement difficulties early on. 


Variant 3 (ten-minute and 24-hour). No time challenges were reported for the first session in this 
variant, although it includes the same depth of content as the first session in Variant 2. It may be the 
case that the juggling ‘spaces’ help to address the difficulty of having the three longer blocks of 
content from each subject in the one session. One teacher, however, reported that juggling 
throughout the lesson was difficult for some students, and that they began to feel shy about struggling 
with it. 
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Conclusion 


Key conclusions 

1. The spaced learning principle is based on relatively strong evidence and this pilot suggests that 
SMART Spaces has evidence of promise. 

2. This pilot study demonstrated that SMART Spaces can feasibly be delivered in English schools. 
Both teachers and pupils appeared to enjoy and engage with the programme. 

Teachers generally reported that they delivered SMART Spaces lessons as prescribed and did 
not make major alterations. 

The small randomised controlled trial provided some preliminary evidence that the most 
promising approach to spaced learning combines both ten-minute and 24-hour spaces. However 
this was a small study and a larger trial is needed to fully understand the impact of the 
programme. 

5. SMART Spaces is ready for a larger RCT to evaluate its impact on GCSE attainment. 


Formative findings 


As indicated earlier, the programme has had two iterations of change. The first changes came after 
the formative findings from the FS, which saw adaptations to the materials, delivery, and training. 
Similarly, after the OS the evaluation and project teams concluded that the ten-minute and 24-hour 
combined version of the programme shows the best evidence of promise for improving science 
attainment. Furthermore, this version is underpinned by evidence from both the neuroscientific and 
cognitive experimental literature. Therefore it has been chosen as the core model for the SMART 
Spaces programme. The final version of the spaced learning programme in presented in Figure 3. 


It is important to emphasise the training element of the programme (SMART Training), particularly in 
the light of a recent review of the effectiveness of secondary science education programmes (Cheung 
et al., 2016). This review highlights that effective secondary science programmes require a strong 
focus on professional development and support for teaching, rather than simple material-focused 
innovations. 


Key recommendations with regard to training are: 


e All teachers that deliver the programme need to be trained by an experienced project team. 
Even teachers who have some experience of spaced learning may not be familiar with the 
specific delivery model that SMART Spaces requires. 

e SMART Training should include a significant practice element where the teachers are 
provided with the opportunity to deliver a spaced learning lesson and receive feedback on 
their performance. 

e One teacher should provide all lessons within the programme, for two reasons: (1) to build 
their confidence in the materials, and (2) to ensure that the programme is completed in the 
allotted time with the minimum disruption to timetabling and teacher availability. 

e The juggling activity should be supported by some training guidance on how to run this 
session successfully. 
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Figure 3: SMART Spaces logic model 
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Interpretation 


Previous literature has shown the benefits of both short spaces (around ten minutes) and medium- 
term spaces (24 hours plus) on the retention of information. Furthermore, some educational practice 
literature shows the benefits of both these kinds of spaces in a real educational setting. However, the 
emerging picture from this research would suggest that there is a benefit to combining both short ten- 
minute and medium-term 24-hour spaces for improving attainment outcomes. Coupled with the fact 
that these lessons can be feasibly delivered in classroom settings makes it a potentially powerful way 
to conduct revision for GCSE science exams. 


As mentioned in the rationale, the aim of this study was to consider how the theory relating to spaced 
learning might translate into a practical model—an optimal spacing pattern—for improving attainment 
outcomes. This has resulted in the ten-minute and 24-hour model underpinning SMART Spaces. The 
theory of change, in terms of the neurological and cognitive changes that occurred as a result of the 
spaces, was not explored. 


However, one interesting finding with regard to the 24-hour variant producing a detrimental effect may 
lead to one hypothesis on the theory of change: the 24-hour variant did not provide any spaces within 
each daily session of learning and thus provided substantial amounts of information in a massed way 
in each session over the three days. Teachers who provided this version commented on the fact that 
this was difficult and tiring for the pupils. Looking at the neuroscientific literature, it has been observed 
that short-term spacing may produce effects on memory through neurochemical refreshment (for 
example, refreshment of proteins such as CREB) and/or the benefits of switching to novel tasks. It 
may be that the mechanism of change was not in operation in the 24-hour version and yet is 
important in producing the effects in spaced learning practice. It may be hypothesised that the short 
spaces provide some neurobiological advantage that can be built on by the 24-hour spaces that were 
also included in the ten-minute and 24-hour combined variant. The hypothesised theory of change, 
therefore, is that proximal changes produced by short spaces are required to build on the distal 
benefits of longer spaces in this way of teaching. In the cognitive psychology literature, 24-hour 
spaces have been shown to be an effective spacing strategy for retention. The learning experience, 
however, may be more difficult as the long session of intake of material without breaks may cause 
fatigue. It would seem that when the ten-minute breaks are combined with these 24-hour spaces, the 
fatigue is reduced and the benefits of 24-hour spaces may be revealed. This is speculation at this 
point: other explanations of the added value of the ten-minute space may be related to the reduction 
in interference between the different subject areas taught. Further neuroscientific and cognitive 
experimental study would be required to investigate the theories of change operating in SMART 
Spaces. 


Another observed pattern was that the variants performed better against the pure control group than 
the ‘slides-only’ group. This suggests that there is some intrinsic benefit in the way the content is 
presented in a high intensity manner. Therefore, it is important that the slides are of high quality and 
updated regularly based on the current curriculum. 


Finally, it is worth noting that both the 24-hour and the ten-minute and 24-hour combined version of 
spaced learning included elements similar to interleaving as they used slightly different content in the 
one session. There are many further nuances to interleaving research (Taylor and Rohrer, 2010) that 
were not explored in this study that would need further investigation. 


There are several limitations to this research. First, the sample size is relatively small for a trial, and 
not adequately powered to detect expected effects, and the number of schools and pupils per 
condition was also small and varied substantially. This variation in numbers per condition, or an 
anomalous school in terms of implementation quality, could have had an undue influence on the effect 
size of the particular variant being delivered. In addition, the measures used, although found to be 
reliable and demonstrating validity, do not provide actual GCSE science scores. Furthermore, this 
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programme was implemented as a revision programme, so it needs additional testing for its 
contribution to initial learning. Finally, this was not a well-controlled RCT, and increased rigour in this 
respect will clarify the effectiveness of SMART Spaces. 


Readiness for trial 


The evaluation team suggest that SMART Spaces is ready for an efficacy trial. The process of design 
and feasibility has provided substantial learning on the feasibility of the programme’s implementation, 
as well as preliminary evidence of its promise for improving science attainment. Furthermore, the 
measure to assess its effect performed well (although actual GCSE scores may be a better outcome 
measure for a larger trial). In addition, there seems to be good reliability and validity to the 
engagement measure—this would be a useful tool for monitoring and assessing implementation in a 
larger trial. 


A contribution to the scalability and replicability of SMART Spaces is the development of a manual. 
The manual will be informed by the findings in this study and help ensure fidelity of delivery going 
forward. Some suggestions for the design and research questions in a larger trial are discussed in the 
next section on future research. 


Another necessary consideration for going towards a bigger trial and potentially greater roll-out of the 
programme is cost. The cost per pupil is £10. This cost is a one off cost to get the teacher trained and 
provide the school with the necessary materials to run the programme for three years. In theory, this 
cost is for the first cohort of pupils, but will also cover subsequent cohorts of pupils being taught by 
the same teacher. The cost per pupil is therefore decreasing with each cohort that goes through, 
however changes in staff, and to the GCSE curriculum, may require refresher packs and training 
sessions for schools. 


Future research and publications 


The evaluation team suggest an efficacy trial at the next stage with further exploration of feasibility 
using both qualitative and quantitative measures (particularly the engagement questionnaire included 
in this study, which can help monitor this significantly related implementation factor). A key 
unanswered question at this point is the cumulative effect of SMART Spaces if used throughout the 
school year. This question could be answered by having a three-arm trial of SMART Spaces with the 
programme occurring at the beginning and end of the final term of the GCSE course compared to 
twice at the end of the course just before exams. This design would provide the opportunity to 
compare the use of ten-minute, 24-hour, and a whole-term space against the current ten-minute and 
24-hour model of SMART Spaces. It is also recommended that the current bespoke measure is used 
as a pre-test, and actual GCSE scores are used as the post-test outcome measure. Some suggested 
research questions for future investigations include: 


1. What are the effects of SMART Spaces on GCSE science attainment? 


2. Does implementing SMART Spaces at several time-points (at the beginning and end of final 
term) have an additional beneficial effect on GCSE science attainment? 


3. Can SMART Spaces be delivered with fidelity (and engagement) in a scaled up version of the 
programme? 


The evaluation team are currently co-producing a SMART Spaces training manual and SMART 
Training programme that will be a useable guide for trainers and teachers in the delivery of the 
SMART Spaces programme. They are also intending to publish some of the results in this report in an 
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international high quality academic journal with potential further papers in science education practice 
journals. 
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Appendix 1: Summary of earlier key studies 


Study type Authors (year) Input Output Outcome 
Cognitive Toppino & Access to nursery Free recall First grade children 
psychology DiGeorge and 1° grade experiments showed a clear 
experiment (1984) children. One to comparing spacing effect, 
(pre-schoolers one testing with massed and whereas pre-school 
and 1° graders) researcher and spaced repetitions | children did not. It is 
child. Testing of 20 | of words with possible that attention 
children individually | nursery and first is consciously 
clearly less cost grade children, of | diverted from 
effective than A different word repeated stimuli in 
classroom based lists (each list massed repetitions. 
study. contained 3 words | This is where ERP 
which were research would be 
repeated ina very useful. 
massed format, 
and 3 repeated in 
a spaced format). 
Review paper Lee & Review paper. 47 articles on Distributed practice 
Genovese effect of improved both 
(1988) distributed acquisition and 
practice on retention of motor 
acquisition and skills. 
retention of motor 
skills. 
Review paper Moss (1996) Review paper. Review of 120 Longer ISls (inter 
articles on study intervals) 
distributed helped verbal 


practice effect, 
across age of 
participant and 
material type 
(verbal 
information, 
intellectual skill, 
motor learning) 


information (e.g. 
spelling) and motor 
learning (e.g. mirror 
tracing) in over 80% 
of studies. Only one 
third of studies 
showed that spaced 
practice helped 
intellectual skill (e.g. 
math computation) 
and half of them 
showed no effect. 


Neuroscience — 
Animals 


Kogan et al. 
(1997) 


Animal research lab 
facilities, raising of 
genetically mutated 
mice. 


Comparison of 
spaced and 
massed learning 
with mice who 
lacked the CREB 
protein ( a protein 
important for 
creation of long 
term memories). 


Spaced training with 
ISIs of 60 minutes 
allowed typical 
memory performance 
of mice who 
otherwise were 
unable to form long 
term memories for 
spatial, contextual 
and social 
information. 


Neuroscience - 
Animals 


Mauelshagen, 
Sherff & 
Carew (1998) 


Animal research lab 
facilities — Aplysia 
(marine molluscs). 


Comparison of 
spaced and 
massed serotonin 
exposure with 
aplysia. Recorded 
intracellular firing 
of synapses using 


Long term synaptic 
facilitation (electrical 
activity recorded up to 
24 hours later) was 
significantly increased 
when serotonin was 
applied in5 x5 
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electrodes. 


minute exposures 
with 15 min intervals, 
compared with 
massed exposure of 
25 min constant 
application. 


Review paper 


Neuroscience - 


Donovan & 
Radosevich 
(1999) 


Menzel, Manz, 


Review paper. 


Animal research lab 


63 articles 
reviewed for wide 
range of tasks. 
Examined effects 
of methodological 
rigour (3 point 
scale), mental 
requirements (low 
or high ‘mental or 
cognitive skills’), 
overall 
complexity, ISI 
(<1 min, 10min - 1 
hr, >1day) and 
retention interval 
(< or > 1 day). 


Massed vs 


The largest effects 
were found in low 
rigour studies with low 
task complexity (e.g. 
typing). The retention 
interval did not show 
any effects. Task 
domain moderated 
the distributed 
practice effect. They 
only examined one 
interaction, ISI and 
task domain; 
Increasing ISI led to 
larger effect sizes in 
verbal tasks like 
verbal discrimination 
and foreign language. 
This formed a U- 
shaped curve, where 
too long of an ISI led 
to a smaller effect. 
Skill based tasks like 
typing or gymnastics 
showed smaller effect 
sizes when intervals 
were increased. 
Spaced conditioning 


Animals Menzel & facilities — spaced learning - _ led to better 
Greggers honeybees. Conditioning of acquisition and 
(2001) honeybees to retention rates, 
show proboscis especially for long 
response for intervals, i.e. late long 
feeding, when term memory, 
different CS assessed at 4 days 
(odours etc.), US | (not only higher rates, 
(strengths of but more stable). A 
sucrose solution) | consolidation process 
and inter-trial takes place when 
intervals (ITI) are | spaced conditioning is 
used. used (they found it 
could be inhibited by 
blocking protein 
synthesis during 
acquisition). 
Review paper. Janiszewski et | Review paper. 97 articles on This study found that 


al. (2003) 


distributed 
practice were 
reviewed. 


verbal vs pictorial 
stimuli, novel vs 
familiar stimuli, 
unimodal vs bimodal 
stimulus presentation, 
structural vs semantic 
cue relationships and 
isolated vs context 
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embedded stimuli did 
not influence effect 
size. Five factors 
influenced effect size 
magnitude: lag 
(longer ISI = larger 
effect), 
meaningfulness of 
stimulus = larger 
effect size than non- 
meaningful, complex 
stimulus (i.e. 
semantically complex) 
led to larger effect 
size than structurally 
complex or simple 
stimuli, intentional 
learning larger effect 
than incidental 
learning, complex 
intervening material 
(i.e. semantically 
complex) led to larger 
effect size than 
structurally complex 
or simple intervening 
material. 


Computational/ 
mathematical 
modelling 


Raajmakers 
(2003) 


Programming of 
computer model of 
memory. 


Tested the SAM 
(Search of 
Associated 
Memory) model 


Claimed that spacing 
effects are due to 
additional contextual 
information stored 
when the material is 
presented ata 
second time point 
(encoding variability). 
This increases the 
chance of the context 
of the testing context 
being similar to the 
study context. 


Neuroscience — 
Animals 


Commins, 
Cunningham, 
Harvey & 
Walsh (2003) 


Animal research 
lab, rats. 


Massed (16 trials) 
vs spaced (4 trials 
per day for 4 
days). Inter trial 
intervals of 10 
seconds. Training 
for a water maze 


Spatial memory was 
impaired in massed 
relative to spaced 
trained group. Worse 
at water maze and 
worse at detecting 
object displacement. 


and object Perhaps more of a lag 
displacement effect, as training was 
task. still multiple trials with 
10s intervals in 
massed condition. 
Review paper. Cepeda et al. Review of 317 Analysis of Only 12 of 217 
(2006) experiments in 184 | spacing vs studies failed to show 
articles. All verbal massed. Analysis | a benefit of spacing 
memory, alltest of | of lag effect over massed study. 


recall. 


(comparison of 
different ISls). 


Lag effect was 
nonmonotonic - 
Increasing ISI 
increases recall, but 
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too long an ISI fora 
given retention period 
reduced recall. ISI 
should increase as 
retention interval 
increases to optimise 
recall. When retention 
interval was less than 
1 minute, optimal ISI 
was also less than 1 
minute. When 
retention interval was 
6 months, at least a 1 
month ISI was 
needed to be optimal. 
For education, the 
practical implication is 
that at least 1 day 
would be needed for 
ISI to be optimal for a 
retention period of 
months. 


Neuroscience - | Van Strien et EEG lab. EEG experiment Larger N400 (index of 

Humans al. (2007) with 22 Dutch memory search) for 
adults. massed than spaced 
Continuous repetitions. LPC (late 
recognition task positive complex - 
(old vs new) matching of template) 
followed by larger massed than 
unexpected free spaced repetitions. 
recall task. 

Spaced practice | Carpenter et Researcher Children givena Children givena 


=—s° grade al. (2009) administered history test and review with ISI of 16 
students. learning sessions then a review of weeks recalled more 
during class time. answers/facts. than children given 
Review of facts the review with ISI of 
was repeated 1 week. Note that 
either 1 week or recall was still very 
16 weeks later. poor for both groups 
Retention test 9 (12% vs 8%) due to 
months later. large retention period. 
Again we see that 
long retention interval 
benefits from longer 
ISI. 
Lab based Cepeda et al. Lab based learning | Experiment 1 — Nonmonotonic effect 
experiment — (2009) experiments over Learning of of ISI. Increases in 
complex task period of up to 12 Swahili-English gap led to sharp 
(language). months with 215 word pairs. ISI| of =| improvement in 


undergraduates. 


5 min to 14 days, 
retention interval 
10 days. 
Experiment 2 — 
Learning of facts 
and objects, ISI of 
20min to 6 
months, retention 
interval 6 months. 


retention, then 
decline. 28 day gap 
was optimal for 6 
month retention 
interval. 


Neuroscience - 
Animals 


Fields (2009) 


Research lab — 
animals. 


Stimulation of 
synapses taken 


The synaptic 
connection, i.e. how it 
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from sections of 
rat hippocampus. 
Studied synaptic 
facilitation after 


fires itself, is 
strengthened. Then 
when it is stimulated 
again, it produces 


spaced and about 2x as much 
massed voltage after 3 bursts 
stimulation (by with 10 min gaps. 
recording This strengthening of 
electrical activity synaptic firing is 
and gene called long term 
presence). potentiation (LTP). 
Found that both 
CREB and a gene 
called zif268 were 
present in synapse 
after spaced 
stimulation - these are 
both associated with 
long term 
potentiation, and 
conversion of short to 
long term memory. 
Spaced practice | Kerfoot (2010) | Design of online Randomised Adaptive spaced 
— Online course on complex | controlled trial — repetition of items led 
learning course material (medical Random to students 
on urology for school level allocation of completing 20% fewer 
medical education) blocks of students | items (i.e. more 
students. to either adaptive | efficient, less 


spaced or 
nonadaptive 


spaced education. 


Adaptive = sent 2 
items daily, if 
correct, then not 
repeated, if 
incorrect, 
repeated 7 days 
later until all 28 
items covered. 
Nonadaptive = 2 
items daily for 40 
days, items 
repeated 10 days 
later. Retention 
lag varied due to 
adaptive period. 


unnecessary 
repetition of known 
items). Comparable 
learning with less 
effort. 


This would work well 
with online course 
with lots of individual 
assessment during 
study — less practical 
in classroom where 
lessons need to be 
tailored to larger 
numbers. 


Spaced practice 
— Complex - 
Native Malay 
speakers 
learning 
English, 
alongside their 
degree. 


Bird (2011) 
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Design of grammar 
and syntax lessons 
for English asa 
second language. 
Administration to 
university students. 


Compared ISIs of 
3 and 14 days for 
retention intervals 
of 7 and 60 days. 
Grammar 
lessons. 


No difference in test 
performance at 7 day 
RI, between 3 and 14 
day ISI. For 60 day 
RI, only 14 day ISI 
showed sustained 
improvement over 
pre-test. Paper 
discusses massed vs 
distributed, but even 
the ‘massed’ was at 3 
day intervals. More of 
a lag effect than 
spacing effect (which 
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is spaced vs totally 
non-spaced). 
Neuroscience - | Mollison & EEG research lab. Comparison of Spaced pairs recalled 
humans Curran (2011) learning of pairs more accurately than 
(noun + picture) massed. Brain 
when presented response more similar 
in spaced (12 to second 
second) or presentation of 
massed format. spaced items than of 
Recorded EEG massed items. This is 
during task — thought to be 
more similar brain | indicative of 
response to increased attention to 
spaced items = P2 when spaced, 
similar levels of because less 
attention leading repetition suppression 
to better recall effect. 
Neuroscience — | Zhang et al. Computational Used Long term synaptic 
animals (2011) modelling of computational facilitation (i.e. 
biochemical modelling to synaptic activity 
cascades in determine optimal | necessary for long 
animals & Animal ISI for serotonin term memory, 
research lab. application to including presence of 
aplysia. CREB) was greater 
Determined that for the enhanced non- 
non-uniform uniform spacing 
applications of protocol over 
serotonin would standard spacing (5x5 
have optimal min ISI) or massed 
effects — ISIs of (25min constant). 
10,10,5 then 30 Basically this expands 
mins. Then tested | on Maulshagen et al. 
on aplysia (1998) by adding the 
synapses. enhanced non- 
uniform spacing 
protocol. 
Neuroscience - | Xue etal. fMRI lab. Adults memorised | If successfully 
Humans (2012) 120 novel faces, learned, stronger 
half massed, half | activation in bilateral 
spaced. Brain fusiform gyrus (area 
activity recorded associated with face 
using fMRI. recognition). Massed 
led to less activity 
because of repetition 
suppression (Same 
face in massed 
suppressed activity). 
Spacing = less 
repetition suppression 
of activity and better 
recognition. 
Neuroscience - | Kramar et al. Animal research Electrical Twice as much long 
Animals (2012) lab. stimulation of rat term potentiation 
hippocampal occurred when bursts 
slices at 10, 30 or | of electrical 
60 min intervals. stimulation were 
administered at 60 
min intervals than at 
10 or 30 min intervals. 
Lab based Jackson, Basic visuospatial Younger vs older | Both young and older 
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experiment with 
neurobiological 
component 
(measurement 
of amyloid 
levels in older 


people). 


Maruff and 
Snyder (2013) 


paired learning, 
plus measurement 
of amyloid levels in 
blood. 


adults. Each 
group did both 
massed and 
spaced learning 
of visuospatial 
information. 
Spacing intervals 
were 15 mins. 
Retention period 
was 24 hours. 


adults showed 
spacing effect. Subset 
of older participants 
showed elevated 
amyloid levels (a 
marker of cognitive 
decline and dementia 
risk). Showed 
decrease of spacing 
effect as amyloid 
levels increased. 


Spaced practice | Kelley and Training of teachers | Spaced learning Spaced learning 
Whatson to administer (3 x 20 minutes of | produced similar long 
(2013) spaced learning stimulation with term memory 
techniques in ISI of 10 mins performance to 
classroom. High distraction) vs massed learning, 
stakes test on massed learning despite significantly 
national curriculum | (4 months typical | less teaching time 
Biology course. teaching). and presence of 
distractor activities. 
Spaced learning is 
claimed to be much 
more time effective, 
i.e. learning per hour 
of instruction. 
Lab based Bell et al. Instruction of Participants All spaced conditions 
experiment — (2014) Swahili-English learned Swahili- showed spacing 
complex — pairs and English word effect. 12 hours with 
learning of new subsequent testing. | pairs. Comparison | sleep and 24 hours 
language. of massed vs with sleep showed 
spaced (12 hours | enhanced spacing 
without sleep) vs | effect over 12 hours 
spaced (12 hours | no sleep. /t is possible 
with sleep) vs that the longer ISI 
spaced 24 hours itself is not solely 
(with sleep). responsible for 
enhanced retention. 
The presence of 
sleep with a shorter 
interval may be just 
as effective. 
Lab based Huff and Basic lab based Investigation of Related to 
experiment Bodner (2014) | learning and the effect of Raajmakers et al. 
memory tasks with | encoding (2003) 
undergraduates. variability (i.e. Bird et al. 1978 found 
different tasks that encoding 
when learning variability did not 
lists) on spacing influence spacing 
effect. effect. This study 
finds that variable 
processing did benefit 
memory, but most 
strongly when both 
tasks and type of 
processing varied 
across study 
experiences. 
Lab based Miles (2014) Teaching of English | Comparison of Spacing effect found, 
experiment — grammar to South massed vs despite the complex 
complex Korean TESOL spaced (ISIs of 1 | nature of the material 
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grammar students. week then 4 (many lab based 

learning. weeks). Retention | experiments are 
interval of 5 simpler recall tasks). 
weeks. 

Spaced practice | Kapler, Teaching of Learned material, | 8 day spaced group 

(undergraduate | Weston and psychology then reviewed scored more highly at 

class) Wlseheart undergraduates — either 1 day or 8 the retention test than 

(2015) classroom days later. 5 week | the 1 day spaced 


environment. 


retention interval. 


group. Benefit of 
longer spacing for 
longer retention 
interval and for 
meaningful/higher 
level material. 
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Figure 3: Summary flow chart of spaced learning research and development in the three areas 
of neuroscience, experimental psychology and educational practice 


Neuroscience 


timulation of animal brains, 
using either electricity or 

serotonin was found to lead 

to stronger long term 

potentiation when short 

intervals of 5 - 15 mins were 
used rather than massed 
stimuilation (e.g. Kogan 
1997, Maulshagen 1998, 

Fields 2009 


Human neuroscientific research 
is severely lacking in this area, 
but Mollison & Curran (2011) 

found EEG evidence of 
increased attention to the second 
presentation of a stimulus when 
it was presented at a 12 second 
delay rather than in a massed 
format, and this was associated 
with better recall. 


More recent work with 
animal brains has 
suggested than non- 
uniform spaces, or longer 
spaces in general, such as 
60 mins lead to stronger 
LTP (Kramar et al. 2012; 
Zhang et al., 2011). 
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Spaced 
Learning 


Lab Based 
Experiments 


Earlier reviews of spaced 
learning experiments 
have found strong 
spacing effects for 
simple recall tasks such 
as verbal memoty (e.g. 
Cepeda et al., 2006; 
Moss 1996) 


The effect of lag on the 
benefit of spacing is 
thought to be 
nonmonotonic. This 
means that as retention 
interval increases, the 
optimal IST also 
increases. However too 
long an ISI for a given 
RI is detrimental 

(Cepeda et al., 2006) 


Lab experiments with more 
meaningful stimuli, such as 
foreign language learning or 
grammar have demonstrated 
that an ISI of multiple weeks 
benefits learning when the 
retention interval is also long 
(weeks or months). See Bell 
et al, (2014) and Miles (2014) 


Spaced 
Practice 
(education 

al setting 


Spaced learning in an 
education setting was 
found to be effective by 
Kelley and Whatson (2013) 
using 10 minute ISIs. This 
strategy was designed on 
the basis of LTP benefits 
from 10 min spacing 


(Fields, 2009) 


Mote recent spaced 
practice in education, at a 
universty level, has found 

that spacing of 8 to 14 
days benefitted retention 
over massed or shorter 
intervals (Bird, 2011; 
Kapler et al., 2015; 
Kerfoot, 2010; 


Adaptive spacing procedures 
(Le. repeating only the 
information which was 
retained poorly the first 
time) have been found to 
benefit learning over non- 
adaptive spacing (Kerfoot, 
2010). Although this works 
well with online courses, this 
may be less practical in a 
classroom setting. 
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Appendix 2: Post-test implementation survey (SMART 
Spaces Engagement Questionnaire) 


Name: School: 
Class: 
Please answer the following questionnaire with how Sironuil Sippel 
much you disagree or agree with the statements none Disagree Agree gy 
: : : Disagree Agree 
based on your experience with Spaced Learning 
| was enthusiastic to try Spaced Learning Oo oO oO oO 
The Spaced learning lessons helped me learn more 
oO oO O oO 
easily than normal lessons 
| think Spaced Learning works well for revision oO oO 
The spaced Learning lessons ran smoothly 
The space activity activities (e.g. juggling) were helpful oO oO 
Spaced learning would work for all classes, not just 
eos Oo oO oO oO 
revision 
| think 3 lessons was enough time for Spaced Learning. oO oO 
The class as a whole enjoyed Spaced learning oO oO 
The teacher was confident at delivering Spaced 
O oO O oO 
Learning 
| felt more motivated to learn during Spaced Learning 
; oO oO O oO 
than in normal classes 
| would be happy to try Spaced learning again in the 
oO oO oO oO 
future 
| think Spaced Learning would also be useful for other 
: O oO O oO 
subjects 
| found the spaced learning lessons tiring oO oO oO oO 
| found the spaced learning lessons fun Oo oO oO oO 
| found the spaced learning lessons helpful for revision oO oO oO oO 
| found Spaced Learning too repetitive oO O O O 
oO Oo oO oO 


| did not enjoy the space activities (e.g. juggling) 
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Multiple Choice Section 


Please circle one answer for each question 


1 How well was the idea behind Spaced learning explained to the class by your teacher? 


a. Very well b. Well c. Poorly d. Very poorly 


2 Was the activity that you did during the spaces juggling, or some other activity? 


a. Juggling b. Some other activity (please specify) 


3 Did the teacher cover all the slides in each lesson or skip over some? 


a. Covered all the slides 
b. Skipped over some slides in some Spaced Learning lessons 


c. Skipped over some slides in every Spaced Learning session 


4 Was Spaced learning delivered to your class 3 days in a row? 


a. Yes 
b. No, there were days in between Spaced Learning lessons 


c. No, we received Spaced learning lessons fewer than 3 times 


5 Was the time period devoted to juggling, or other activity, the same each day? 
a. Yes b. No 
6 How were the science subjects divided across the lessons? 


a. |had one science subject per day of the project 


b. 1|had all 3 sciences on each day of the project 


7 How often did your class take part in the juggling activity? 


@ 


My class juggled multiple times throughout each lesson 


a 


My class juggled only at the end of the lesson 
My class performed a different spacing activity throughout each lesson 


d. My class performed a different spacing activity only at the end of each lesson 
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Table 11: Responses to the engagement questionnaire administered to pupils at post-test 


Strongly Strongly 
Disagree | Disagree | Agree Agree 
% % % % 
| was enthusiastic to try Spaced Learning 5.5% 9.4% 63.3% 21.9% 
The Spaced learning lessons helped me 6.7% 23.5% 48.6% 21.2% 
learn more easily than normal lessons 
| think Spaced Learning works well for 4.7% 14.4% 51.0% 30.0% 
revision 
The spaced Learning lessons ran 5.9% 17.6% 56.5% 20.0% 
smoothly 
The space activity activities (e.g. juggling) 8.2% 23.4% 41.8% 26.6% 
were helpful 
Spaced learning would work for all | 12.2% 39.2% 34.1% 14.5% 
classes, not just revision 
| think 3 lessons was enough time for] 13.3% 45.3% 33.6% 7.8% 
Spaced Learning.SL 
The class as a whole enjoyed Spaced 5.6% 20.6% 48.4% 25.4% 
learning 
The teacher was confident at delivering 3.1% 5.9% 50.4% 40.6% 
Spaced Learning 
| felt more motivated to learn during 7.0% 25.0% 41.0% 27.0% 
Spaced Learning than in normal classes 
| would be happy to try Spaced learning 7.1% 11.4% 44.9% 36.6% 
again in the future 
| think Spaced Learning would also be 7.9% 18.6% 47.8% 25.7% 
useful for other subjects 
| found the spaced learning lessons tiring 18.9% 48.0% 24.4% 8.7% 
| found the spaced learning lessons fun 5.1% 12.9% 55.9% 26.2% 
| found the spaced learning lessons 5.1% 19.6% 49.8% 25.5% 
helpful for revision 
| found Spaced Learning too repetitive 15.0% 44.9% 29.9% 10.2% 
| did not enjoy the space activities (e.g. | 41.6% 36.9% 15.3% 6.3% 


juggling) 
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Table 12: Responses to the fidelity questionnaire administered to pupils at post-test 


Question Response % 
Very Well 41.0% 
How well was the idea behind Spaced | Well 50.6% 
learning explained to the class by Poorly 48% 
your teacher? 
Very Poorly 3.6% 
Was the activity that you did during Juggling 91.5% 
the spaces juggling, or some other Other 8.5% 
activity? 
Covered all slides 90.2% 
Did the teacher cover all the slides in | sxinned some slides in some lessons 8.5% 
each lesson or skip over some? : ; ; 
Skipped some slides in every lesson 1.2% 
Yes 81.1% 
Was Spaced learning delivered to No, there were days in between lessons 18.5% 
your class 3 days in a row? No, we received Spaced learning fewer than 3 A% 
times 
Was the time period devoted to Yes 86.4% 
juggling, or other activity, the same No 13.6% 
each day? 
How were the science subjects One science per day 62.1% 
divided across the lessons? All three per day 37.9% 
Class juggled multiple times throughout each 68.5% 
lesson 
Class juggled only at the end of the lesson 27.0% 
ow offen did ae take part in Class performed a different spacing activity 3.3% 
the juggling activity? throughout the lesson 
Class performed a different spacing activity only at 1.2% 


the end of the lesson 
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Appendix 3: Information Sheet and Memorandum of 
Understanding for Schools 


School Agreement to participate in a feasibility study of the Spaced Learning Project. 
School Name and Email ----------------------------------------------------2 non nnnn nnn nnn nnn nnn 
Name and email of Lead for Project in school--------------------------------------------------- 

Aims of the Feasibility Study 


The aim of this feasibility study is to develop, and evaluate the materials and resources required for 
deployment of teaching based on Spaced Learning principles in Y9/10 classes. We hope that the 
outcomes of this test phase will enable us to develop a method (a programme manual) and materials 
(for teaching the programme). The results of this feasibility study will contribute towards our 
understanding of what works in terms of improving retention of science subject knowledge with Y9/10 
classes and will be disseminated to schools in England and possibly elsewhere. Ultimately, we hope 
the results from these test schools will equip us to better understand Spaced Learning and support 
those wishing to use this teaching and learning style as an additional tool for helping their pupils to 
retain the subject knowledge required for science attainment at GCSE. 


Rationale 


The 'Spaced Learning’ intervention is a method and materials for teaching that is inspired by models 
of learning from psychology and neuroscience evidence that indicate spaces between presentation of 
information improves memory and recall of information. Currently, Spaced Learning takes the form of 
three bursts of teaching that each cover the same content in different ways, with intervals of 
distraction activity (e.g. juggling, using clay). The specifics of the model will be explored in feasibility 
project. For example, there is debate over the optimal length of time between the teaching bursts. 
However, the intervention will have to be practical for delivery in schools and hence this project will 
test delivery in real English schools. This project will attempt to explore these issues through a series 
of research questions. 


The research questions 


In this study the focus will be on design, feasibility and implementation of a ‘spaced learning’ 
programme and will therefore test slightly different versions of the programme against each other. 
There are three main questions regarding the best version of the programme: 


What version shows best potential efficacy? 
What version has the best potential for implementation in the classroom? 
What is the best version for ‘going to scale’ i.e., rolling out across many schools?. 


The Methodology 
Part 1 - Training 


Initially, your teachers will be required to attend meetings and workshops run by the project team to 
familiarise yourselves with how SL works in practice. 
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It is envisaged that the project team will work closely with your schools (approx.12-15 schools) in 
order to ensure that you are given the intervention and are aware of who will participate, what 
materials you will use, what activities you will try in the spaces, how you will deliver the lesson. 


Part 2 - Programme delivery 


Schools are asked to attempt to implement different versions of the programme variants in your 
respective classes (potentially including control groups which will carry on with typical practice.) 


Part 3 - Programme data collation for analysis 


Your school staff and pupils will be required to provide qualitative and quantitative data for the 
evaluation team (Queen’s University Belfast) to analyse (further details below). Your data will be 
imperative in adapting the existing materials to further develop the manual for use in future delivery 
and research of the SLP. 


Your participation will help to produce the final output in the form of a ‘spaced learning’ manual 
featuring: a literature review; logic model (explicating mechanisms of outcome change); 
implementation data; training guide; and programme content. 


Your participation 


Your school has been contacted by the project team based on our funder’s preferences that schools 
have significant proportions of pupils eligible for pupil premium and are representative of the 
population of English state schools in terms of Ofsted ratings, GCSE scores, etc. 

You also have interest in Spaced Learning principles and a range of familiarity with the concept but 
have not been implementing spaced learning with those classes participating, 


Use of Data 


Pupils’ test responses and all pupil data will be treated with the strictest confidence. Named data will 
be matched with the National Pupil Database and shared within the Evaluation team (QUB). 


Aggregated and anonymised data will be shared with the Project Delivery team (HTSA) EEF and the 
UK Data Archive. No individual school or pupil will be identified in any report arising from this initial 
feasibility study or from the final RCT for the SLP 


Responsibilities 
The PROJECT TEAM (HTSA) will: 


Organise and deliver training for schools in Spaced learning intervention, making sure teachers 
understand what is to be done in classes and how SL lessons should be organised 

Provide PowerPoint and other materials to be used 

Be the first point of contact for any questions about the delivery of SL lessons and the use of 
materials. 

Provide on- going support to schools 

Send out regular updates of the progress of the project in its entirety through a newsletter 


The EVALUATION TEAM (QUB) will: 
Randomly allocate schools to variations of the programme 
Collect pre and post GCSE test data from pupils 
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Organise and deliver focus groups with pupils and teachers to gather data on the feasibility of 
implementing a ‘spaced learning’ programme 

Carry out classroom observations of spaced learning lessons 

Be the first point of contact for any questions about the opt-out consent forms, workshops 
Collate and analyse data from these focus groups and lesson observations 

Disseminate research findings 


The SCHOOL will 


Ensure the shared understanding and support of all school staff, including the Governing Body, for the 
project and personnel involved 

Ensure attendance at meetings /workshops organised by the project team 

Consent to random allocation to variants of the programme and commit to the lifetime of the project 
(September 2015 September 2016) for the successful completion of the initial feasibility study. 
Remember that some schools or classes could be in the control group and in this case the school will 
be asked not to deliver spaced learning in any format to these students until after the agreed testing. 
Consent to data collection within the school, in terms of focus groups with pupils and staff (including 
leadership ) 

Consent to data collection within the school, in terms of classroom observations and science 
measures based on GCSE Science questions. 

Implement interventions as requested based on evaluation team allocation, including if necessary 
releasing 2-3 staff so they can attend all meetings and workshops 

Agree to changing the way lessons are delivered for the purposes of fulfilling the SL lesson delivery 
Share requested school level data with the Project team 

Inform all parents /carers about the project and collect opt out consent forms 

Be a point of contact for parents /carers seeking more information on the project 

Share requested pupil level data with the project team 

Inform parents /carers about any additional data collection — information provided by Project team 


We commit to the Second Feasibility Study ( OS ) of the Spaced Learning Project 


School 
Name 


Headteacher 
Name 


Headteacher Signature Date 


Names and Emails of Participant staff 


Chair of Governing Body Date 


Thank you for agreeing to take part in this research. Please sign both copies, retaining one and 
returning the second copy to Farhana Zaman at Hallam Teaching School Alliance Notre Dame 
High School, Fulwood Road, Sheffield, S10 3BT 
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Appendix 4: Outline of topics, marks and GCSE paper 
sources for the Science Measure 


Subject Topic Marks Source paper Total 
marks 
Biology Keeping Healthy — Balanced diet 1 AQA-BL1FP-Jun14 14 
Short Metabolic rate 1 AQA-BL1HP-Jun13 
Answer : 
and White blood cells 1 AQA-BL1HP-Jan13 
Multiple Vaccines 2 AQA-BL1HP-Jan13 
Choice Antibiotics & resistance 2 AQA-BL1HP-Jun14 
Growing bacteria 1 AQA-BL1FP-Jun13 
Nervous system — synapses 1 AQA-BL1FP-Jan12 
Fertility hormones 1 AQA-BL1HP-Jan13 
Water & salt control 2 AQA-BLIFP-Jan13 
Plant hormones 1 AQA-BL1iFP-Jan12 
Plant hormones 1 AQA-BL1iFP-Jan12 
Chemistry | Basic atomic model electron | 1 AQA- 12 
Short configuration CHILDHOod1HP- 
Answer Jani2 
and Types of bonding 1 AQchildhoodCH1FP- 
Multiple Jun14 
Choice Reduction fl CHILDHOodA- 
CH1FP-Jan12 
Electrolysis 1 AQA-CH1FP-Jun14 
Phytomining & Bioleaching 1 AQA-CH1FP-Jun14 
Properties of metals 2 AQA-CH1FP-Jan12 
Hydrocarbons 1 AQA-CH1FP-Jan12 
Hydrocarbons 1 AQA-CH1FP-Jan12 
Cracking alkanes 1 AQA-CH1FP-Jun14 
Fractional distillation 2 AQA-CH1FP-Jan12 
Physics Infrared 1 AQA-PH1FP-Jan13 13 
Pisatee Change of state 1 AQA-PH1FP-Jan13 
and Convection, conduction, evaporation 2 AQA-PH1FP-Jun13 
Multiple Rate of evaporation 1 AQA-PH1FP-Jan13 
Choice Energy transfer by design 1 AQA-PH1HP-Jun12 
U-value & insulation 1 AQA-PH1HP-Jan13 
Types of energy 3 AQA-PH1FP-Jun14 
Power stations 2 AQA-PH1HP-Jun12 
Fossil fuels 1 AQA-PH1FP-Jun13 
Biology Growing bacteria 6 AQA-BL1FP-Jan13 6 
Extended 
Answer 
Chemistry | Fractional distillation 6 AQA-CH1FP-Jun13 6 
Extended 
Answer 
Physics Va—uums - Insulation 6 AQA-PH1FP-Jun13 6 
Extended 


Answer 
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Appendix 5: Example SMART Spaces slide 


Power stations connected to your house: : 


* Voltage is stepped up for transmission with less energy loss. 

* National grid loses power as cables get hot. 

* Voltage is stepped down for household appliances to use. 
*Pylons are ugly but the cables are cooler than undergrounlarges 
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Appendix 6: Time lapsed between pre-test and post-test for 


each school in OS 


School Time lapsed between pre-test and post-test 

(days) 
A 

63 
B 

37 
Cc 

65 
D 

62 
E 

63 
F 

63 
G 

63 
H 

63 
| 

49 
J 

63 
K 

63 
L 
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Appendix 7: Mixed-effects ML regression 


Mixed-effects ML regression (IV = pupil engagement, DV = Post-test total score) , Number of pupils n 
= 223 (from 10 min, 24 hour, 10min/24 hr, and no spaces control groups). Group variable: School 
Number of schools = 9 


Pre-test 0.85 0.04 <0.01 
Engagement with SMART SPACES | 0.12 0.05 0.01 
Constant 1.65 2.20 0.45 
-2 Log Likelihood -671.72 


Mixed-effects ML regression (IV = retention interval, DV = Post-test total score) , Number of pupils n = 
313 (all groups). Group variable: School 


Number of schools = 10 


coef. s.e p= 
Pre-test 0.89 0.03 <0.01 
Retention Interval -0.02 0.07 0.75 
Constant 5.80 1.68 <0.01 
-2 Log Likelihood -945.83 
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Appendix 8: Pre, post and gain scores for all variants 


Pre-test Post-test Gain 

Variant Score Mean SD Mean SD 
10 Min Total score 27.91 9.09 30.82 9.82 2.91 
Short answer 21.25 6.25 23.52 6.59 2.27 
Extended answer 6.66 4.07 7.30 4.41 0.64 
24Hour Total score 25.04 12.81 27.30 14.64 2.26 
Short answer 20.72 9.23 23.19 10.81 2.47 
Extended answer 4.32 4.13 4.12 4.62 -0.20 
No Spaces Total score 21.51 8.54 23.99 8.89 2.48 
Short answer 17.62 6.05 19.35 6.60 1.73 
Extended answer 3.89 3.56 4.63 3.72 0.75 
10Min/24Hour | Total score 23.93 10.59 27.55 12.22 3.62 
Short answer 19.93 8.09 22.41 8.69 2.47 
Extended answer 4.00 3.65 5.14 4.30 1.14 
No Total score 25.28 7.16 26.82 9.80 1.54 
spaces/materials | Short answer 19.06 541 | 20.32 6.94 1.26 
Extended answer 6.23 2.88 6.50 3.90 0.27 
No ___ | Total score 23.62 6.51 | 24.07 7.52 0.45 
pr oie shorbanswer 17.64 4.81 | 18.34 5.38 0.7 
L Extended answer 5.98 2.80 5.73 3.38 | -0.25 
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